Jul 16 21:44:46.118953 ip-10-0-139-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:44:46.118967 ip-10-0-139-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:44:46.118977 ip-10-0-139-156 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:44:46.119303 ip-10-0-139-156 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:44:56.356694 ip-10-0-139-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:44:56.356715 ip-10-0-139-156 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1c1a0d6ab1f142ea9859f6ee814b7508 --
Jul 16 21:46:57.099826 ip-10-0-139-156 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:46:57.491631 ip-10-0-139-156 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:46:57.491631 ip-10-0-139-156 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:46:57.491631 ip-10-0-139-156 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:46:57.491631 ip-10-0-139-156 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:46:57.491631 ip-10-0-139-156 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:46:57.493459 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.493369    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:46:57.498843 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498827    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498853    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498858    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498862    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498865    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498867    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498871    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498874    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498876    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498879    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498881    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498884    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498887    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498890    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498892    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498895    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:57.498890 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498898    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498901    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498904    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498907    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498910    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498912    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498915    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498917    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498920    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498922    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498925    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498927    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498930    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498932    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498935    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498937    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498940    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498943    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498945    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498948    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:57.499278 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498950    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498953    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498955    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498958    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498960    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498962    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498965    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498969    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498971    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498974    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498976    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498979    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498981    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498984    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498987    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498990    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498992    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498995    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.498998    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:57.499789 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499002    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499006    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499009    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499012    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499015    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499018    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499020    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499023    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499026    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499028    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499030    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499033    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499036    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499039    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499041    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499044    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499047    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499050    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499053    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499056    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:57.500253 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499058    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499061    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499063    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499066    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499068    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499072    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499075    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499080    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499083    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499085    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499088    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499091    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499094    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499096    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499099    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499101    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499103    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499106    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499108    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:57.500754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499111    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499113    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499117    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499121    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499123    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499847    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499854    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499857    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499859    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499862    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499865    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499868    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499871    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499874    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499876    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499879    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499881    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499884    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499887    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499890    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:57.501210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499893    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499896    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499899    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499903    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499907    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499910    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499913    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499916    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499919    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499922    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499924    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499928    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499931    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499934    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499937    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499940    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499943    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499945    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499949    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:57.501754 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499953    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499955    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499958    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499961    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499963    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499966    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499968    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499971    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499974    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499977    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499979    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499982    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499984    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499987    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499989    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499992    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499995    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.499997    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500000    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500003    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:57.502228 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500005    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500008    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500010    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500012    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500015    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500017    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500019    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500022    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500024    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500027    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500029    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500032    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500035    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500037    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500040    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500042    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500045    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500047    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500049    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500053    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:57.502736 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500055    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500057    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500060    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500062    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500065    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500069    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500072    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500075    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500078    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500080    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500083    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500085    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500087    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500090    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500092    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500095    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500099    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500101    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500104    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500106    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:57.503221 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500109    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500111    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500113    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500116    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500119    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500536    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500541    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500561    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500565    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500569    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500573    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500576    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500579    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500582    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500586    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500589    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500593    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500596    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500599    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:57.503751 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500601    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500605    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500607    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500610    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500613    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500616    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500618    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500621    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500623    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500626    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500629    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500631    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500634    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500638    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500640    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500643    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500645    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500648    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500650    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:57.504218 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500653    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500655    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500658    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500660    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500663    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500665    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500668    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500670    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500673    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500675    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500678    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500681    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500684    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500686    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500689    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500691    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500694    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500696    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500699    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500701    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:57.504688 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500704    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500707    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500709    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500712    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500714    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500717    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500719    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500723    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500727    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500729    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500732    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500734    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500737    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500739    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500742    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500744    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500747    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500749    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500751    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:57.505199 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500755    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500757    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500760    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500763    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500766    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500768    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500771    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500774    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500776    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500779    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500781    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500784    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500786    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500789    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500791    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500794    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500796    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500799    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500802    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500804    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:57.505682 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500807    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500809    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500811    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500814    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500816    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500819    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.500822    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500895    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500902    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500911    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500917    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500934    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500937    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500942    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500946    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500949    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500952    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500956    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500959    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500965    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500968    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500971    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500974    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:46:57.506194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500977    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500979    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500983    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500986    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500989    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500992    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500996    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.500999    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501002    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501005    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501010    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501013    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501016    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501019    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501023    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501026    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501031    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501034    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501037    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501040    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501043    2562 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501045    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501050    2562 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501053    2562 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:46:57.506760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501056    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501059    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501062    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501065    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501069    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501072    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501075    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501077    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501080    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501083    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501086    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501088    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501091    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501094    2562 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501098    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501101    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501104    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501107    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501111    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501114    2562 flags.go:64] FLAG: --help="false"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501117    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501120    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501123    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:46:57.507419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501126    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501129    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501133    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501136    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501138    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501141    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501144    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501147    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501150    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501153    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501156    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501159    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501161    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501167    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501170    2562 flags.go:64] FLAG: --lock-file=""
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501173    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501175    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501178    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501184    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501186    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501189    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501192    2562 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501195    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501198    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:46:57.508356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501201    2562 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501203    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501208    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501212    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501216    2562 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501219    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501222    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501225    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501228    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501231    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501234    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501238    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501245    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501248    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501251    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501254    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501257    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501263    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501265    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501268    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501271    2562 flags.go:64] FLAG: --port="10250"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501276    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501279    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fadfe74b02ecaef9"
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501282    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:46:57.509004 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501285    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501288    2562 flags.go:64] FLAG: --register-node="true"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501291    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501294    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501297    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501300    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501303    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501307    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501310    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501312    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501315    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501318    2562 flags.go:64] FLAG: --runonce="false"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501322    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501325    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501328    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501331    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501333    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501337    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501340    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501342    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501345    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501348    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501351    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501354    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501357    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:46:57.509605 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501359    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501362    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501367    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501370    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501374    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501378    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501381    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501383    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501386    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501389    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501392    2562 flags.go:64] FLAG: --v="2"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501396    2562 flags.go:64] FLAG: --version="false"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501400    2562 flags.go:64] FLAG: --vmodule=""
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501404    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501407    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501528    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501532    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501535    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501538    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501543    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501560    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501564    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501568    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:57.510246 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501572    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501574    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501577    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501580    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501583    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501587    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501590    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501593    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501596    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501599    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501601    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501604    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501606    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501610    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501613    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501615    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501618    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501620    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501624    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:57.510837 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501627    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501630    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501633    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501636    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501638    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501641    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501643    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501646    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501648    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501653    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501656    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501658    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501661    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501664    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501667    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501669    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501672    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501675    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501677    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501680    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:57.511306 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501683    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501685    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501688    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501690    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501693    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501695    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501699    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501701    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501704    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501706    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501709    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501711    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501714    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501716    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501719    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501722    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501724    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501727    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501729    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:57.511828 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501733    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501737    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501741    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501744    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501746    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501749    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501751    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501754    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501757    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501759    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501762    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501765    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501767    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501770    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501773    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501775    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501778    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501780    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501783    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501786    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:57.512297 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501789    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501791    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501794    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501796    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501798    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501801    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501804    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501806    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501809    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501811    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501814    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501817    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.501819    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.501829    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.510119    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.510135    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:46:57.512813 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.510154    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.510160    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510231    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510237    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510242    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510246    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510249    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510252    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510255    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510258    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510261    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510264    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510267    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510270    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510273    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510276    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510278    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510281    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510283    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:57.513210 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510286    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510288    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510291    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510294    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510296    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510299    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510301    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510304    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510307    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510309    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510313    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510315    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510318    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510320    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510323    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510325    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510328    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510330    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510333    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:57.513735 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510335    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510338    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510340    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510342    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510345    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510347    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510350    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510352    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510355    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510357    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510360    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510363    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510365    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510368    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510370    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510372    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510376    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510378    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510381    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510383    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:57.514202 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510386    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510388    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510390    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510393    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510395    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510398    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510400    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510403    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510406    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510408    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510411    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510413    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510416    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510418    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510420    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510423    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510425    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510428    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510431    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510433    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:57.514704 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510436    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510438    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510442    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510446    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510448    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510451    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510454    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510457    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510460    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510464    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510468    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510470    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510474    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510476    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510479    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510482    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510484    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510487    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510490    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:57.515192 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510493    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510495    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510497    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510500    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.510505    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510635    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510641    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510644    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510647    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510650    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510652    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510657    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510660    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510663    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:57.515691 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510666    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510668    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510671    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510675    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510678    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510680    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510683    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510685    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510688    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510691    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510693    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510696    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510698    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510700    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510703    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510705    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510708    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510711    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510714    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:57.516043 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510717    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510719    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510722    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510724    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510726    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510729    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510731    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510734    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510736    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510738    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510741    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510743    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510745    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510748    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510750    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510753    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510755    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510759    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510762    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510764    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:57.516507 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510766    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510769    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510771    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510774    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510777    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510779    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510781    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510784    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510786    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510789    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510791    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510794    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510796    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510799    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510802    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510806    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510808    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510811    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510814    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510816    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:57.517014 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510819    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510822    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510824    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510827    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510830    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510832    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510835    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510837    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510840    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510842    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510846    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510848    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510851    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510853    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510856    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510858    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510861    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510863    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510866    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510869    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:57.517493 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510873    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510876    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510879    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510881    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510883    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510886    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510888    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510891    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510893    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510895    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:57.510898    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.510903    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.511015    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.514498    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:46:57.518107 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.516136    2562 server.go:1019] "Starting client certificate rotation"
Jul 16 21:46:57.518452 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.516233    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:46:57.518452 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.516261    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:46:57.518510 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.518499    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:46:57.520015 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.519999    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:46:57.529077 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.529059    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:46:57.531560 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.531531    2562 log.go:25] "Validated CRI v1 image API"
Jul 16 21:46:57.533515 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.533498    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:46:57.536025 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.536004    2562 fs.go:135] Filesystem UUIDs: map[5de7de38-2a22-4666-aac8-83d6772b89b5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a3ba6acd-9707-43ad-803d-e73142635c52:/dev/nvme0n1p3]
Jul 16 21:46:57.536118 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.536025    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:46:57.542714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.542565    2562 manager.go:216] Machine: {Timestamp:2026-07-16 21:46:57.541122873 +0000 UTC m=+0.323020711 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101105 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec234025e116f20c34f8b60f875f557d SystemUUID:ec234025-e116-f20c-34f8-b60f875f557d BootID:1c1a0d6a-b1f1-42ea-9859-f6ee814b7508 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4c:a3:b1:0d:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4c:a3:b1:0d:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:e0:29:26:33:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:46:57.542714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.542704    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:46:57.542832 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.542810    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:46:57.545497 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.545479    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:46:57.545682 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.545659    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:46:57.545813 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.545688    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:46:57.546363 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.546353    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:46:57.546392 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.546366    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:46:57.546392 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.546373    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:46:57.547048 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.547037    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:46:57.548256 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.548245    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:46:57.551098 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551087    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:46:57.551217 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551207    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:46:57.551245 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551239    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:46:57.551272 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551249    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:46:57.551272 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551262    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:46:57.551272 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551270    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:46:57.551419 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551278    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:46:57.551666 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551653    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:46:57.551712 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.551672    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:46:57.554899 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.554878    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:46:57.556735 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.556722    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:46:57.556791 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.556753    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:46:57.557815 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557802    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557819    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557825    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557831    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557844    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557852    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557860    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557866    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557873    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557879    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:46:57.557883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557887    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:46:57.558297 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557893    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:46:57.558297 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557917    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:46:57.558297 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.557922    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:46:57.561126 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.561113    2562 server.go:1268] "Started kubelet"
Jul 16 21:46:57.561261 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.561233    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:46:57.561582 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.561357    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:46:57.561582 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.561429    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:46:57.562041 ip-10-0-139-156 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:46:57.562837 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.562755    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:46:57.562837 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.562802    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:46:57.563042 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.562874    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:46:57.563042 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.562959    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:46:57.563467 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.563456    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:46:57.566868 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.566846    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:46:57.569132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.569111    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgqgc"
Jul 16 21:46:57.569749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.569734    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:46:57.569810 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.569739    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:46:57.570078 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.569892    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:57.570310 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570286    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:46:57.570386 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570321    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:46:57.570386 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570286    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:46:57.570491 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570456    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:46:57.570491 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570465    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:46:57.570739 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570720    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:46:57.570830 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570739    2562 factory.go:55] Registering systemd factory
Jul 16 21:46:57.570830 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.570764    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:46:57.571038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.571017    2562 factory.go:153] Registering CRI-O factory
Jul 16 21:46:57.571038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.571038    2562 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:46:57.571175 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.571061    2562 factory.go:103] Registering Raw factory
Jul 16 21:46:57.571175 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.571071    2562 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:46:57.578330 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.570466    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-156.ec2.internal.18c2e36331556b15  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-156.ec2.internal,UID:ip-10-0-139-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-156.ec2.internal,},FirstTimestamp:2026-07-16 21:46:57.561086741 +0000 UTC m=+0.342984581,LastTimestamp:2026-07-16 21:46:57.561086741 +0000 UTC m=+0.342984581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-156.ec2.internal,}"
Jul 16 21:46:57.578330 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.574734    2562 manager.go:349] Starting recovery of all containers
Jul 16 21:46:57.580417 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.580387    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgqgc"
Jul 16 21:46:57.581766 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.581740    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:46:57.582274 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.581952    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:46:57.587480 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.587317    2562 manager.go:354] Recovery completed
Jul 16 21:46:57.592537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.592523    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.594448 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.594432    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.594525 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.594462    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.594525 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.594473    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.595014 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595002    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:46:57.595014 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595012    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:46:57.595109 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595027    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:46:57.595109 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595011    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.595780 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595764    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.595844 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595793    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.595844 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.595807    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.596340 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.596330    2562 policy_none.go:49] "None policy: Start"
Jul 16 21:46:57.596369 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.596345    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:46:57.596369 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.596357    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:46:57.596874 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.596793    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-156.ec2.internal.18c2e363335277b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-156.ec2.internal,UID:ip-10-0-139-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-156.ec2.internal,},FirstTimestamp:2026-07-16 21:46:57.594447796 +0000 UTC m=+0.376345637,LastTimestamp:2026-07-16 21:46:57.594447796 +0000 UTC m=+0.376345637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-156.ec2.internal,}"
Jul 16 21:46:57.597527 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.597513    2562 policy_none.go:47] "Start"
Jul 16 21:46:57.638092 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638076    2562 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:46:57.638189 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.638108    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:46:57.638189 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638117    2562 server.go:87] "Starting device plugin registration server"
Jul 16 21:46:57.638365 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638353    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:46:57.638416 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638367    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:46:57.638504 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638486    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:46:57.638609 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638595    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:46:57.638609 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.638608    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:46:57.639080 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.639039    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:46:57.639164 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.639110    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:57.738965 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.738922    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.740263 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.739907    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.740263 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.739940    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.740263 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.739950    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.740263 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.739974    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.743078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.743025    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:46:57.744436 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.744413    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:46:57.744530 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.744447    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:46:57.744530 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.744478    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:46:57.744530 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.744512    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:46:57.746914 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.746890    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:57.747767 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.747752    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.747831 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.747775    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-156.ec2.internal\": node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:57.815278 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.815248    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:57.845372 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.845336    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"]
Jul 16 21:46:57.845446 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.845413    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.846968 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.846952    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.847067 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.846982    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.847067 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.846994    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.848226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848214    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.848374 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.848424 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848387    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.848956 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848938    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.849017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848960    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.849017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848971    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.849017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848978    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.849017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848986    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.849017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.848991    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.850154 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.850139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.850193 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.850181    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:57.850836 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.850819    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:57.850920 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.850845    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:57.850920 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.850855    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:57.871769 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.871742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6fe804e0022788129898c1b4a906153-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal\" (UID: \"c6fe804e0022788129898c1b4a906153\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.871873 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.871774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fe804e0022788129898c1b4a906153-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal\" (UID: \"c6fe804e0022788129898c1b4a906153\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.871873 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.871799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80f21118e6e677b8e438ca527e429584-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.ec2.internal\" (UID: \"80f21118e6e677b8e438ca527e429584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.873140 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.873123    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-156.ec2.internal\" not found" node="ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.877083 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.877064    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-156.ec2.internal\" not found" node="ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.915681 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:57.915654    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:57.972564 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.972526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80f21118e6e677b8e438ca527e429584-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.ec2.internal\" (UID: \"80f21118e6e677b8e438ca527e429584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.972713 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.972574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6fe804e0022788129898c1b4a906153-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal\" (UID: \"c6fe804e0022788129898c1b4a906153\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.972713 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.972592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fe804e0022788129898c1b4a906153-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal\" (UID: \"c6fe804e0022788129898c1b4a906153\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.972713 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.972647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6fe804e0022788129898c1b4a906153-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal\" (UID: \"c6fe804e0022788129898c1b4a906153\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.972713 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.972635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80f21118e6e677b8e438ca527e429584-config\") pod \"kube-apiserver-proxy-ip-10-0-139-156.ec2.internal\" (UID: \"80f21118e6e677b8e438ca527e429584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:57.972713 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:57.972655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6fe804e0022788129898c1b4a906153-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal\" (UID: \"c6fe804e0022788129898c1b4a906153\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:58.016145 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.016084    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.116875 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.116842    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.176343 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.176316    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:58.180384 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.180175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:58.217868 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.217834    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.318428 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.318343    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.420866 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.420832    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.457420 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.457393    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:58.515934 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.515894    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:46:58.516481 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.516119    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:46:58.516481 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.516120    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:46:58.521243 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.521212    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.585898 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.585819    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:41:58 +0000 UTC" deadline="2028-01-20 13:27:18.100409699 +0000 UTC"
Jul 16 21:46:58.585898 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.585850    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13263h40m19.514562865s"
Jul 16 21:46:58.622251 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:58.622210    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-156.ec2.internal\" not found"
Jul 16 21:46:58.623755 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.623737    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:58.635914 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.635895    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:58.670624 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.670592    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:58.686331 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.686296    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:46:58.687693 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.687674    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal"
Jul 16 21:46:58.699465 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.699443    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:46:58.750383 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:58.750353    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f21118e6e677b8e438ca527e429584.slice/crio-9c329378389083c4539be2acf64d0f5626b1ce56c72637c9143efe082d7b063d WatchSource:0}: Error finding container 9c329378389083c4539be2acf64d0f5626b1ce56c72637c9143efe082d7b063d: Status 404 returned error can't find the container with id 9c329378389083c4539be2acf64d0f5626b1ce56c72637c9143efe082d7b063d
Jul 16 21:46:58.750946 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:46:58.750920    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fe804e0022788129898c1b4a906153.slice/crio-2a15487f7102fa06d1bea633ab0f06e7118875a2dad913f57ec1bb71f4f9f6c3 WatchSource:0}: Error finding container 2a15487f7102fa06d1bea633ab0f06e7118875a2dad913f57ec1bb71f4f9f6c3: Status 404 returned error can't find the container with id 2a15487f7102fa06d1bea633ab0f06e7118875a2dad913f57ec1bb71f4f9f6c3
Jul 16 21:46:58.756537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:58.756523    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:46:59.412091 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.412056    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:59.553860 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.553827    2562 apiserver.go:52] "Watching apiserver"
Jul 16 21:46:59.563581 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.563547    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:46:59.565724 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.565694    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4rs4r","kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal","openshift-dns/node-resolver-98t4h","openshift-image-registry/node-ca-6q4k4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal","openshift-multus/multus-pxb2g","openshift-network-diagnostics/network-check-target-sxb4t","openshift-network-operator/iptables-alerter-cvf8x","kube-system/global-pull-secret-syncer-h25ht","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc","openshift-cluster-node-tuning-operator/tuned-rj5qs","openshift-multus/multus-additional-cni-plugins-kffsw","openshift-multus/network-metrics-daemon-kbs6s","openshift-ovn-kubernetes/ovnkube-node-cccgg"]
Jul 16 21:46:59.567274 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.567249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:46:59.567368 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.567334    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:46:59.568843 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.568822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.569866 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.569845    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.571118 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.571094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.571701 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.571678    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:46:59.572568 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.572223    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cbvld\""
Jul 16 21:46:59.572568 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.572273    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.572568 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.572447    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.572568 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.572486    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.572854 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.572669    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.572942 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.572904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.573499 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.573470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.573675 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.573651    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r9k2x\""
Jul 16 21:46:59.573993 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.573974    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.575943 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.575924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.576049 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.575993    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:46:59.578286 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.577393    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.578286 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.577423    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n866j\""
Jul 16 21:46:59.578286 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.577600    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:46:59.578286 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.577676    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:46:59.579393 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.579363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.580313 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.579578    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:46:59.580313 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.579921    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.580313 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.580149    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:46:59.580523 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.580370    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.580879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kxj2v\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581098    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-host\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-node-log\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581369    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysconfig\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581525    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrlc\" (UniqueName: \"kubernetes.io/projected/15ef1262-6afb-4406-bfb4-8401f7ba21c0-kube-api-access-jvrlc\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-ovnkube-config\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581644    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581652    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-ovnkube-script-lib\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-systemd\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-sys\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15ef1262-6afb-4406-bfb4-8401f7ba21c0-tmp\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-var-lib-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.582294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-cni-bin\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-cni-netd\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581903    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rvzk\""
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-modprobe-d\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581950    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.581991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-etc-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-systemd-units\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-systemd\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-log-socket\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-run-ovn-kubernetes\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmx88\" (UniqueName: \"kubernetes.io/projected/145430b6-2caf-411f-a744-e590d9551eee-kube-api-access-nmx88\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-var-lib-kubelet\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-tuned\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e344137-20c3-47a9-97c0-a800bf47e475-tmp-dir\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4bv\" (UniqueName: \"kubernetes.io/projected/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-kube-api-access-xs4bv\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysctl-d\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-run\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:46:59.583344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.582977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-lib-modules\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-host\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-run-netns\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-env-overrides\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/145430b6-2caf-411f-a744-e590d9551eee-ovn-node-metrics-cert\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-kubernetes\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysctl-conf\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-serviceca\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-ovn\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-kubelet\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-slash\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e344137-20c3-47a9-97c0-a800bf47e475-hosts-file\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.584101 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.583569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2q5\" (UniqueName: \"kubernetes.io/projected/4e344137-20c3-47a9-97c0-a800bf47e475-kube-api-access-ls2q5\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.584780 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.584549    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbw5s\""
Jul 16 21:46:59.584780 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.584763    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:46:59.584905 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.584890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.585325 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.585308    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.585489 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.585450    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:46:59.585702 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.585630    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-knd79\""
Jul 16 21:46:59.586500 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.586481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.587417 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.587397    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:46:59.587798 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.587779    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:46:59.587966 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.587947    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:46:59.588160 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.588144    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cvdww\""
Jul 16 21:46:59.588494 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.588476    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:46:59.588843 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.588824    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l4phh\""
Jul 16 21:46:59.590198 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.590177    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:46:59.590427 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.590411    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:46:59.592674 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.592650    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:46:59.624442 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.624422    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjss2"
Jul 16 21:46:59.635646 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.635604    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjss2"
Jul 16 21:46:59.671729 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.671667    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-socket-dir-parent\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-cni-bin\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-etc-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-systemd\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-var-lib-kubelet\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e344137-20c3-47a9-97c0-a800bf47e475-tmp-dir\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52a5b129-e482-4486-a543-a7ae3a011c36-konnectivity-ca\") pod \"konnectivity-agent-4rs4r\" (UID: \"52a5b129-e482-4486-a543-a7ae3a011c36\") " pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-cni-bin\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-etc-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-run-ovn-kubernetes\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-systemd\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-var-lib-kubelet\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.684471 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684459    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-run-ovn-kubernetes\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f37685f-898d-4bda-935d-85652bdd317b-iptables-alerter-script\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-device-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-host\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-netns\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-run-netns\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-host\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/145430b6-2caf-411f-a744-e590d9551eee-ovn-node-metrics-cert\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e344137-20c3-47a9-97c0-a800bf47e475-tmp-dir\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-kubernetes\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysctl-conf\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-serviceca\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-kubernetes\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-run-netns\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.685226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-slash\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.684949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysctl-conf\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-slash\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e344137-20c3-47a9-97c0-a800bf47e475-hosts-file\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-serviceca\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e344137-20c3-47a9-97c0-a800bf47e475-hosts-file\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2q5\" (UniqueName: \"kubernetes.io/projected/4e344137-20c3-47a9-97c0-a800bf47e475-kube-api-access-ls2q5\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-cnibin\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685487    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk95\" (UniqueName: \"kubernetes.io/projected/3f37685f-898d-4bda-935d-85652bdd317b-kube-api-access-rqk95\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xgd\" (UniqueName: \"kubernetes.io/projected/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-kube-api-access-c5xgd\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-node-log\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-kubelet\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685599    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f37685f-898d-4bda-935d-85652bdd317b-host-slash\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-node-log\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysconfig\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-etc-kubernetes\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-ovnkube-config\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-ovnkube-script-lib\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-systemd\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685922    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysconfig\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-k8s-cni-cncf-io\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.685982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-cni-bin\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cnibin\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-var-lib-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-systemd\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-cni-netd\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-var-lib-openvswitch\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-modprobe-d\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.686892 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-systemd-units\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-tuned\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-os-release\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1796c5f4-1783-4838-9e5e-bbc9b8de961f-cni-binary-copy\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-modprobe-d\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-conf-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-log-socket\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-systemd-units\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmx88\" (UniqueName: \"kubernetes.io/projected/145430b6-2caf-411f-a744-e590d9551eee-kube-api-access-nmx88\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-log-socket\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-cni-netd\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-ovnkube-script-lib\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4bv\" (UniqueName: \"kubernetes.io/projected/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-kube-api-access-xs4bv\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-ovnkube-config\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-sys-fs\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsll\" (UniqueName: \"kubernetes.io/projected/9b33f558-7844-420c-b60a-fbeef1574933-kube-api-access-zjsll\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.687714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-system-cni-dir\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-os-release\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysctl-d\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-run\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-lib-modules\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52a5b129-e482-4486-a543-a7ae3a011c36-agent-certs\") pod \"konnectivity-agent-4rs4r\" (UID: \"52a5b129-e482-4486-a543-a7ae3a011c36\") " pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.686989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-hostroot\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687029    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-run\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-env-overrides\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-sysctl-d\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-daemon-config\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pjr\" (UniqueName: \"kubernetes.io/projected/1796c5f4-1783-4838-9e5e-bbc9b8de961f-kube-api-access-55pjr\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-ovn\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-kubelet\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-host\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6pl\" (UniqueName: \"kubernetes.io/projected/10a2d502-339d-4f01-83ee-966860bc7a48-kube-api-access-xl6pl\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-host-kubelet\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-system-cni-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-multus-certs\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/145430b6-2caf-411f-a744-e590d9551eee-run-ovn\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bec7cf3-d1a3-4497-b829-932fbc703c4a-dbus\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/145430b6-2caf-411f-a744-e590d9551eee-env-overrides\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-host\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrlc\" (UniqueName: \"kubernetes.io/projected/15ef1262-6afb-4406-bfb4-8401f7ba21c0-kube-api-access-jvrlc\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-lib-modules\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-cni-multus\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bec7cf3-d1a3-4497-b829-932fbc703c4a-kubelet-config\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-socket-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-sys\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15ef1262-6afb-4406-bfb4-8401f7ba21c0-tmp\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.689214 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.687997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-cni-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.689962 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.688027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-registration-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.689962 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.688038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15ef1262-6afb-4406-bfb4-8401f7ba21c0-sys\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.689962 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.689457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15ef1262-6afb-4406-bfb4-8401f7ba21c0-etc-tuned\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.689962 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.689700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/145430b6-2caf-411f-a744-e590d9551eee-ovn-node-metrics-cert\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.690323 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.690291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15ef1262-6afb-4406-bfb4-8401f7ba21c0-tmp\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.696418 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.696399    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:46:59.696517 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.696424    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:46:59.696517 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.696439    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:59.696650 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.696521    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:00.19648507 +0000 UTC m=+2.978382920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:59.698946 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.698925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4bv\" (UniqueName: \"kubernetes.io/projected/db43cfec-8a7b-4fc6-8370-c6f8dae02a69-kube-api-access-xs4bv\") pod \"node-ca-6q4k4\" (UID: \"db43cfec-8a7b-4fc6-8370-c6f8dae02a69\") " pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.699059 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.699006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2q5\" (UniqueName: \"kubernetes.io/projected/4e344137-20c3-47a9-97c0-a800bf47e475-kube-api-access-ls2q5\") pod \"node-resolver-98t4h\" (UID: \"4e344137-20c3-47a9-97c0-a800bf47e475\") " pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.700589 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.700566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmx88\" (UniqueName: \"kubernetes.io/projected/145430b6-2caf-411f-a744-e590d9551eee-kube-api-access-nmx88\") pod \"ovnkube-node-cccgg\" (UID: \"145430b6-2caf-411f-a744-e590d9551eee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.701012 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.700993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrlc\" (UniqueName: \"kubernetes.io/projected/15ef1262-6afb-4406-bfb4-8401f7ba21c0-kube-api-access-jvrlc\") pod \"tuned-rj5qs\" (UID: \"15ef1262-6afb-4406-bfb4-8401f7ba21c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.749755 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.749687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal" event={"ID":"c6fe804e0022788129898c1b4a906153","Type":"ContainerStarted","Data":"2a15487f7102fa06d1bea633ab0f06e7118875a2dad913f57ec1bb71f4f9f6c3"}
Jul 16 21:46:59.750930 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.750891    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal" event={"ID":"80f21118e6e677b8e438ca527e429584","Type":"ContainerStarted","Data":"9c329378389083c4539be2acf64d0f5626b1ce56c72637c9143efe082d7b063d"}
Jul 16 21:46:59.788949 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.788918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52a5b129-e482-4486-a543-a7ae3a011c36-konnectivity-ca\") pod \"konnectivity-agent-4rs4r\" (UID: \"52a5b129-e482-4486-a543-a7ae3a011c36\") " pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.788955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f37685f-898d-4bda-935d-85652bdd317b-iptables-alerter-script\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.788985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-device-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-netns\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-cnibin\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789132 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk95\" (UniqueName: \"kubernetes.io/projected/3f37685f-898d-4bda-935d-85652bdd317b-kube-api-access-rqk95\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xgd\" (UniqueName: \"kubernetes.io/projected/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-kube-api-access-c5xgd\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-kubelet\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f37685f-898d-4bda-935d-85652bdd317b-host-slash\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-device-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-etc-kubernetes\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-k8s-cni-cncf-io\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-cni-bin\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cnibin\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f37685f-898d-4bda-935d-85652bdd317b-host-slash\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-os-release\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1796c5f4-1783-4838-9e5e-bbc9b8de961f-cni-binary-copy\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789445 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-kubelet\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-conf-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-cni-bin\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-sys-fs\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsll\" (UniqueName: \"kubernetes.io/projected/9b33f558-7844-420c-b60a-fbeef1574933-kube-api-access-zjsll\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.789547    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-system-cni-dir\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-os-release\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52a5b129-e482-4486-a543-a7ae3a011c36-agent-certs\") pod \"konnectivity-agent-4rs4r\" (UID: \"52a5b129-e482-4486-a543-a7ae3a011c36\") " pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.789640    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:47:00.289621205 +0000 UTC m=+3.071519035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-netns\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f37685f-898d-4bda-935d-85652bdd317b-iptables-alerter-script\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-etc-kubernetes\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52a5b129-e482-4486-a543-a7ae3a011c36-konnectivity-ca\") pod \"konnectivity-agent-4rs4r\" (UID: \"52a5b129-e482-4486-a543-a7ae3a011c36\") " pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.789960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-k8s-cni-cncf-io\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-hostroot\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-daemon-config\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pjr\" (UniqueName: \"kubernetes.io/projected/1796c5f4-1783-4838-9e5e-bbc9b8de961f-kube-api-access-55pjr\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-sys-fs\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-conf-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6pl\" (UniqueName: \"kubernetes.io/projected/10a2d502-339d-4f01-83ee-966860bc7a48-kube-api-access-xl6pl\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-system-cni-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-system-cni-dir\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-system-cni-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-multus-certs\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bec7cf3-d1a3-4497-b829-932fbc703c4a-dbus\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790011    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-os-release\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-cni-multus\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bec7cf3-d1a3-4497-b829-932fbc703c4a-kubelet-config\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-socket-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-os-release\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.790484 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-cni-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bec7cf3-d1a3-4497-b829-932fbc703c4a-dbus\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-registration-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-cni-dir\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790193    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-socket-dir-parent\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-var-lib-cni-multus\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bec7cf3-d1a3-4497-b829-932fbc703c4a-kubelet-config\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.789636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-socket-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-cnibin\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cnibin\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-host-run-multus-certs\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-hostroot\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-daemon-config\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.791403 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790833    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1796c5f4-1783-4838-9e5e-bbc9b8de961f-cni-binary-copy\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.792096 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b33f558-7844-420c-b60a-fbeef1574933-registration-dir\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.792096 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.790848    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:59.792096 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:46:59.790897    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:00.290884126 +0000 UTC m=+3.072781955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:59.792096 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.790894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1796c5f4-1783-4838-9e5e-bbc9b8de961f-multus-socket-dir-parent\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.792096 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.791479    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.793033 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.793014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52a5b129-e482-4486-a543-a7ae3a011c36-agent-certs\") pod \"konnectivity-agent-4rs4r\" (UID: \"52a5b129-e482-4486-a543-a7ae3a011c36\") " pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.799128 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.799103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xgd\" (UniqueName: \"kubernetes.io/projected/f73652f1-9b2c-4387-b0a7-5d90a83e69c7-kube-api-access-c5xgd\") pod \"multus-additional-cni-plugins-kffsw\" (UID: \"f73652f1-9b2c-4387-b0a7-5d90a83e69c7\") " pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:46:59.801046 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.800966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6pl\" (UniqueName: \"kubernetes.io/projected/10a2d502-339d-4f01-83ee-966860bc7a48-kube-api-access-xl6pl\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:46:59.802470 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.802447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsll\" (UniqueName: \"kubernetes.io/projected/9b33f558-7844-420c-b60a-fbeef1574933-kube-api-access-zjsll\") pod \"aws-ebs-csi-driver-node-d8rlc\" (UID: \"9b33f558-7844-420c-b60a-fbeef1574933\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.803215 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.803191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk95\" (UniqueName: \"kubernetes.io/projected/3f37685f-898d-4bda-935d-85652bdd317b-kube-api-access-rqk95\") pod \"iptables-alerter-cvf8x\" (UID: \"3f37685f-898d-4bda-935d-85652bdd317b\") " pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.806537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.806516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pjr\" (UniqueName: \"kubernetes.io/projected/1796c5f4-1783-4838-9e5e-bbc9b8de961f-kube-api-access-55pjr\") pod \"multus-pxb2g\" (UID: \"1796c5f4-1783-4838-9e5e-bbc9b8de961f\") " pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.885801 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.885757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rj5qs"
Jul 16 21:46:59.893205 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.893183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-98t4h"
Jul 16 21:46:59.902957 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.902935    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6q4k4"
Jul 16 21:46:59.909736 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.909714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:46:59.917647 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.917616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxb2g"
Jul 16 21:46:59.926470 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.926416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cvf8x"
Jul 16 21:46:59.940194 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.940175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc"
Jul 16 21:46:59.945019 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.944998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:46:59.952727 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:46:59.952706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kffsw"
Jul 16 21:47:00.294486 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.294388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:00.294486 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.294444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:00.294486 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.294479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294606    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294614    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294616    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294679    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:47:01.294662781 +0000 UTC m=+4.076560608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294705    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:01.294684799 +0000 UTC m=+4.076582629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294629    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:00.294733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294730    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:00.295009 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.294775    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:01.294763448 +0000 UTC m=+4.076661282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:00.353722 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.353694    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145430b6_2caf_411f_a744_e590d9551eee.slice/crio-201d2f3f5b0e380e7f29f6eac3e9627707e671f3ebe60601a215268bcb3ee61f WatchSource:0}: Error finding container 201d2f3f5b0e380e7f29f6eac3e9627707e671f3ebe60601a215268bcb3ee61f: Status 404 returned error can't find the container with id 201d2f3f5b0e380e7f29f6eac3e9627707e671f3ebe60601a215268bcb3ee61f
Jul 16 21:47:00.384992 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.384963    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b33f558_7844_420c_b60a_fbeef1574933.slice/crio-9c4bd11e2003555a73fe848e628d4bcf124e5681929a98083958c1c7a1372d6e WatchSource:0}: Error finding container 9c4bd11e2003555a73fe848e628d4bcf124e5681929a98083958c1c7a1372d6e: Status 404 returned error can't find the container with id 9c4bd11e2003555a73fe848e628d4bcf124e5681929a98083958c1c7a1372d6e
Jul 16 21:47:00.385912 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.385873    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f37685f_898d_4bda_935d_85652bdd317b.slice/crio-8b22069f8557f20efc782cabecc2d74b38bd4bd970318ec7fe18edd9e92394b1 WatchSource:0}: Error finding container 8b22069f8557f20efc782cabecc2d74b38bd4bd970318ec7fe18edd9e92394b1: Status 404 returned error can't find the container with id 8b22069f8557f20efc782cabecc2d74b38bd4bd970318ec7fe18edd9e92394b1
Jul 16 21:47:00.386767 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.386743    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ef1262_6afb_4406_bfb4_8401f7ba21c0.slice/crio-2ca5d1026e2aba889579bd350f35c145a2ac6ea5bb0710ac374f30144c5d0b14 WatchSource:0}: Error finding container 2ca5d1026e2aba889579bd350f35c145a2ac6ea5bb0710ac374f30144c5d0b14: Status 404 returned error can't find the container with id 2ca5d1026e2aba889579bd350f35c145a2ac6ea5bb0710ac374f30144c5d0b14
Jul 16 21:47:00.388081 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.388047    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb43cfec_8a7b_4fc6_8370_c6f8dae02a69.slice/crio-dda96abc22665e1e9350380a636ac2520b0174e37d455bfeb8a7cbe2c910ca26 WatchSource:0}: Error finding container dda96abc22665e1e9350380a636ac2520b0174e37d455bfeb8a7cbe2c910ca26: Status 404 returned error can't find the container with id dda96abc22665e1e9350380a636ac2520b0174e37d455bfeb8a7cbe2c910ca26
Jul 16 21:47:00.389541 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.389326    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e344137_20c3_47a9_97c0_a800bf47e475.slice/crio-3e8e26ef0d7543dac591f7a7cfaa49541ff1d4d10f296cbb7015df9da4542fca WatchSource:0}: Error finding container 3e8e26ef0d7543dac591f7a7cfaa49541ff1d4d10f296cbb7015df9da4542fca: Status 404 returned error can't find the container with id 3e8e26ef0d7543dac591f7a7cfaa49541ff1d4d10f296cbb7015df9da4542fca
Jul 16 21:47:00.390219 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:00.390147    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1796c5f4_1783_4838_9e5e_bbc9b8de961f.slice/crio-d676d13fb1a63a194c2ad66771b98569d1f8a1983377b412879dcd817c3130c3 WatchSource:0}: Error finding container d676d13fb1a63a194c2ad66771b98569d1f8a1983377b412879dcd817c3130c3: Status 404 returned error can't find the container with id d676d13fb1a63a194c2ad66771b98569d1f8a1983377b412879dcd817c3130c3
Jul 16 21:47:00.638240 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.637894    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:00 +0000 UTC" deadline="2028-03-04 01:04:47.293543463 +0000 UTC"
Jul 16 21:47:00.638240 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.638110    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14307h17m46.655439724s"
Jul 16 21:47:00.745233 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.745200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:00.745397 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:00.745343    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:00.754276 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.754242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cvf8x" event={"ID":"3f37685f-898d-4bda-935d-85652bdd317b","Type":"ContainerStarted","Data":"8b22069f8557f20efc782cabecc2d74b38bd4bd970318ec7fe18edd9e92394b1"}
Jul 16 21:47:00.755634 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.755600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-98t4h" event={"ID":"4e344137-20c3-47a9-97c0-a800bf47e475","Type":"ContainerStarted","Data":"3e8e26ef0d7543dac591f7a7cfaa49541ff1d4d10f296cbb7015df9da4542fca"}
Jul 16 21:47:00.756912 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.756890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6q4k4" event={"ID":"db43cfec-8a7b-4fc6-8370-c6f8dae02a69","Type":"ContainerStarted","Data":"dda96abc22665e1e9350380a636ac2520b0174e37d455bfeb8a7cbe2c910ca26"}
Jul 16 21:47:00.758224 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.758176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerStarted","Data":"992ed57365b1dcb2e8e13b98107b088efebf935dacbcfa63782d058a46a01271"}
Jul 16 21:47:00.759440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.759414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxb2g" event={"ID":"1796c5f4-1783-4838-9e5e-bbc9b8de961f","Type":"ContainerStarted","Data":"d676d13fb1a63a194c2ad66771b98569d1f8a1983377b412879dcd817c3130c3"}
Jul 16 21:47:00.760556 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.760517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rj5qs" event={"ID":"15ef1262-6afb-4406-bfb4-8401f7ba21c0","Type":"ContainerStarted","Data":"2ca5d1026e2aba889579bd350f35c145a2ac6ea5bb0710ac374f30144c5d0b14"}
Jul 16 21:47:00.761855 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.761794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc" event={"ID":"9b33f558-7844-420c-b60a-fbeef1574933","Type":"ContainerStarted","Data":"9c4bd11e2003555a73fe848e628d4bcf124e5681929a98083958c1c7a1372d6e"}
Jul 16 21:47:00.763166 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.763136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rs4r" event={"ID":"52a5b129-e482-4486-a543-a7ae3a011c36","Type":"ContainerStarted","Data":"aa6082419a1bf6a3fee463f193e8e5d211d5b3ef3d15c965ab68d81d4ecfd7ec"}
Jul 16 21:47:00.764472 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.764443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"201d2f3f5b0e380e7f29f6eac3e9627707e671f3ebe60601a215268bcb3ee61f"}
Jul 16 21:47:00.766523 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:00.766496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal" event={"ID":"80f21118e6e677b8e438ca527e429584","Type":"ContainerStarted","Data":"2bf00aec4ae03c3e31c485cdeacc369e11d3dbe21c1c8bbdc72b50b71d048c39"}
Jul 16 21:47:01.296839 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.296799    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:01.303845 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.303812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:01.304017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.303864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:01.304017 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.303911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:01.304147 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304067    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:01.304147 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304127    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:47:03.304106887 +0000 UTC m=+6.086004729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:01.304562 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304539    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:01.304666 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304602    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:03.304585674 +0000 UTC m=+6.086483507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:01.304745 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304708    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:01.304745 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304729    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:01.304745 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304741    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:01.304901 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.304778    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:03.304766857 +0000 UTC m=+6.086664688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:01.639388 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.639289    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:00 +0000 UTC" deadline="2028-02-27 15:44:36.410031312 +0000 UTC"
Jul 16 21:47:01.639388 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.639335    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14177h57m34.770701478s"
Jul 16 21:47:01.756799 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.756760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:01.757003 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.756969    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:01.757520 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:01.757490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:01.757693 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:01.757670    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:02.745539 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:02.744997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:02.745539 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:02.745144    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:03.322349 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:03.322310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:03.322560 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:03.322359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:03.322560 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:03.322390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:03.322560 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322516    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:03.322764 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322608    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:47:07.322572891 +0000 UTC m=+10.104470723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:03.322764 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322684    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:03.322764 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322720    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:07.322707496 +0000 UTC m=+10.104605325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:03.322945 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322805    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:03.322945 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322825    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:03.322945 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322838    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:03.322945 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.322873    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:07.32286162 +0000 UTC m=+10.104759453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:03.746225 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:03.745467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:03.746225 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.745659    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:03.746225 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:03.746057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:03.746225 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:03.746141    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:04.744773 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:04.744727    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:04.744972 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:04.744863    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:05.745979 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:05.745216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:05.745979 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:05.745371    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:05.745979 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:05.745463    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:05.745979 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:05.745550    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:06.745865 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:06.745821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:06.746069 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:06.745966    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:07.356586 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:07.356552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:07.356701 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:07.356610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:07.356812 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.356795    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:07.356855 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.356825    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:07.356855 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.356843    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:07.356925 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.356856    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:07.356925 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.356872    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:15.356850982 +0000 UTC m=+18.138748810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:07.356925 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.356891    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:15.356879806 +0000 UTC m=+18.138777634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:07.358030 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:07.357365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:07.358030 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.357471    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:07.358030 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.357522    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:47:15.357507076 +0000 UTC m=+18.139404904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:07.746375 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:07.746302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:07.746798 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:07.746306    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:07.746798 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.746413    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:07.746798 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:07.746520    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:08.745644 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:08.745611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:08.745802 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:08.745750    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:09.747493 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:09.747454    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:09.747951 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:09.747627    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:09.748290 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:09.748110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:09.748290 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:09.748232    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:10.745522 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:10.745488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:10.745708 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:10.745609    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:11.747575 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:11.747529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:11.747983 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:11.747696    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:11.747983 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:11.747755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:11.747983 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:11.747857    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:12.745702 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:12.745668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:12.745875 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:12.745782    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:13.745055 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:13.745018    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:13.745477 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:13.745166    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:13.745477 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:13.745221    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:13.745477 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:13.745365    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:14.744734 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:14.744698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:14.745017 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:14.744815    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:15.417073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:15.417139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:15.417167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417182    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417243    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:47:31.417226672 +0000 UTC m=+34.199124504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417277    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417294    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417308    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417320    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417339    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:31.417322225 +0000 UTC m=+34.199220051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:15.417430 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.417355    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:31.417344533 +0000 UTC m=+34.199242360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:15.747277 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:15.747210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:15.747401 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.747327    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:15.747401 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:15.747381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:15.747484 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:15.747468    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:16.744660 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:16.744625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:16.745074 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:16.744776    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:17.745996 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:17.745524    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:17.745996 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:17.745633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:17.745996 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:17.745656    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:17.745996 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:17.745772    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:18.745155 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:18.745127    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:18.745355 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:18.745254    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:19.746930 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:19.745767    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:19.746930 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:19.745909    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:19.746930 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:19.745959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:19.746930 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:19.746061    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:20.745417 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.745262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:20.745512 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:20.745488    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:20.813981 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.813953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rj5qs" event={"ID":"15ef1262-6afb-4406-bfb4-8401f7ba21c0","Type":"ContainerStarted","Data":"97529fd58ff980c7f24ab8df3f6167e753451c4de82dbd24a0caa3c644052b3f"}
Jul 16 21:47:20.815342 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.815321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc" event={"ID":"9b33f558-7844-420c-b60a-fbeef1574933","Type":"ContainerStarted","Data":"f91dc2e415aad69b2710a26355d7c9c519216c271a61aae7af548f3ed437e6a3"}
Jul 16 21:47:20.816534 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.816513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rs4r" event={"ID":"52a5b129-e482-4486-a543-a7ae3a011c36","Type":"ContainerStarted","Data":"f8416eb2fa0ecd70978c1ad9457d81b1b084a0bbcc375134e19cd53fbfa436f4"}
Jul 16 21:47:20.818538 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.818519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"4d69ccda7d4f03a62a7159425fa763c395ef6b5352b85829e1bf2c0637873248"}
Jul 16 21:47:20.818634 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.818601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"f3c8aa99bbeec1808e9cc5db711d2706b11e90aa4913e2de1db54c0a74f373ac"}
Jul 16 21:47:20.818689 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.818654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"91d0216e2ddd3140b1eff959619fc5cb005eee232c053101968c32909cccd5fa"}
Jul 16 21:47:20.819960 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.819940    2562 generic.go:356] "Generic (PLEG): container finished" podID="c6fe804e0022788129898c1b4a906153" containerID="9c1953d323a41a9bafdabb792d3964d5e33c551056450787a88ed18de4ee2cae" exitCode=0
Jul 16 21:47:20.820045 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.820012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal" event={"ID":"c6fe804e0022788129898c1b4a906153","Type":"ContainerDied","Data":"9c1953d323a41a9bafdabb792d3964d5e33c551056450787a88ed18de4ee2cae"}
Jul 16 21:47:20.821251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.821231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-98t4h" event={"ID":"4e344137-20c3-47a9-97c0-a800bf47e475","Type":"ContainerStarted","Data":"ee54a6d0c5f845c66b63ba85f914b515014eeffd7048ec0e3475cbd2b01e2f32"}
Jul 16 21:47:20.822492 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.822473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6q4k4" event={"ID":"db43cfec-8a7b-4fc6-8370-c6f8dae02a69","Type":"ContainerStarted","Data":"cc894f179f0be0cc180ca7b3a66f2952f217060eef9af44adf47ddb8fae0c9d6"}
Jul 16 21:47:20.823890 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.823864    2562 generic.go:356] "Generic (PLEG): container finished" podID="f73652f1-9b2c-4387-b0a7-5d90a83e69c7" containerID="10fea5a46213128e6bd78af9a703a4800de0d9934637061a4876d35c0cee3a92" exitCode=0
Jul 16 21:47:20.823961 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.823928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerDied","Data":"10fea5a46213128e6bd78af9a703a4800de0d9934637061a4876d35c0cee3a92"}
Jul 16 21:47:20.825465 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.825448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxb2g" event={"ID":"1796c5f4-1783-4838-9e5e-bbc9b8de961f","Type":"ContainerStarted","Data":"91d4df83bb8d18614e73eeec2dac61c98f199d83d848f10ba8c625aeeb9c74e4"}
Jul 16 21:47:20.833462 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.833424    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-156.ec2.internal" podStartSLOduration=21.833403999 podStartE2EDuration="21.833403999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:00.784017318 +0000 UTC m=+3.565915169" watchObservedRunningTime="2026-07-16 21:47:20.833403999 +0000 UTC m=+23.615301850"
Jul 16 21:47:20.852669 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.852635    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rj5qs" podStartSLOduration=2.94600594 podStartE2EDuration="22.852622168s" totalImagesPullingTime="19.906616228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:20.833139506 +0000 UTC m=+23.615037355" watchObservedRunningTime="2026-07-16 21:47:20.852622168 +0000 UTC m=+23.634520006"
Jul 16 21:47:20.868885 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.868848    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-98t4h" podStartSLOduration=2.967366346 podStartE2EDuration="22.868831445s" totalImagesPullingTime="19.901465099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:20.852383413 +0000 UTC m=+23.634281262" watchObservedRunningTime="2026-07-16 21:47:20.868831445 +0000 UTC m=+23.650729294"
Jul 16 21:47:20.884834 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.884796    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4rs4r" podStartSLOduration=8.2654985 podStartE2EDuration="22.884784192s" totalImagesPullingTime="14.619285692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:20.88443372 +0000 UTC m=+23.666331570" watchObservedRunningTime="2026-07-16 21:47:20.884784192 +0000 UTC m=+23.666682039"
Jul 16 21:47:20.884925 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.884906    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6q4k4" podStartSLOduration=2.978504974 podStartE2EDuration="22.884900537s" totalImagesPullingTime="19.906395563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:20.868927678 +0000 UTC m=+23.650825525" watchObservedRunningTime="2026-07-16 21:47:20.884900537 +0000 UTC m=+23.666798385"
Jul 16 21:47:20.906143 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:20.906109    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pxb2g" podStartSLOduration=2.943278512 podStartE2EDuration="22.90609543s" totalImagesPullingTime="19.962816918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:20.905636055 +0000 UTC m=+23.687533904" watchObservedRunningTime="2026-07-16 21:47:20.90609543 +0000 UTC m=+23.687993277"
Jul 16 21:47:21.028096 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.027888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:21.098920 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.098888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:21.199890 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.199858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:21.745598 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.745507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:21.745809 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:21.745652    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:21.745809 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.745694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:21.745809 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:21.745791    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:21.829355 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.829313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cvf8x" event={"ID":"3f37685f-898d-4bda-935d-85652bdd317b","Type":"ContainerStarted","Data":"0a1235f3cf69c22ffd2cb75919fa19aa5db50b3c9c34b7586dd9c312755e36b0"}
Jul 16 21:47:21.833267 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.833230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"9506dc73e85bde68dde5da135f0914df4468d3f63d26e1aca05b1c9a130e715a"}
Jul 16 21:47:21.833404 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.833313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"25de6344022961adf4d5adda7a3bc20869ce4474ee689f2e3d88c79260546c28"}
Jul 16 21:47:21.833404 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.833374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"22008fc44a49395a107668792fad2a24038f135256f64963fba663197a29d953"}
Jul 16 21:47:21.851805 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.851759    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cvf8x" podStartSLOduration=9.240214837 podStartE2EDuration="23.851741043s" totalImagesPullingTime="14.611526206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:21.851461387 +0000 UTC m=+24.633359236" watchObservedRunningTime="2026-07-16 21:47:21.851741043 +0000 UTC m=+24.633638892"
Jul 16 21:47:21.976122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:21.976039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:22.044741 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.044647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:22.538398 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.538377    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:47:22.651360 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.651141    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:47:22.538393069Z","UUID":"7f7301f2-328b-4f5e-9fd1-6a8aa4c73969","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:47:22.652915 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.652871    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:47:22.653052 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.652925    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:47:22.745247 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.745178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:22.745395 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:22.745282    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:22.840352 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.840311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc" event={"ID":"9b33f558-7844-420c-b60a-fbeef1574933","Type":"ContainerStarted","Data":"f619746843b36b0af8f56942ed7793e436acdee9a96346659a172d5561387553"}
Jul 16 21:47:22.842456 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.842427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal" event={"ID":"c6fe804e0022788129898c1b4a906153","Type":"ContainerStarted","Data":"082a7dabb858860cd5363267042a7b09973eaebb36695b7c99c7df2795ffb8ef"}
Jul 16 21:47:22.863115 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:22.863067    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-156.ec2.internal" podStartSLOduration=23.863052737 podStartE2EDuration="23.863052737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:22.86260525 +0000 UTC m=+25.644503097" watchObservedRunningTime="2026-07-16 21:47:22.863052737 +0000 UTC m=+25.644950587"
Jul 16 21:47:23.182798 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.182772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:47:23.183321 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.183294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:47:23.212352 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.212330    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:47:23.212968 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.212948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4rs4r"
Jul 16 21:47:23.351537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.351502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:23.745356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.745325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:23.745559 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:23.745446    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:23.745559 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.745510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:23.745691 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:23.745645    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:23.848625 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:23.848584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"2929bbf17d6c35b9b5f9d788875d7d9a36889ceaaa83ba3c0d3ea78d9f4174d0"}
Jul 16 21:47:24.744691 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:24.744661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:24.744880 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:24.744782    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:24.958381 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:24.958339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:25.056836 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.056759    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:25.745162 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.744978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:25.745327 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.745052    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:25.745327 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:25.745246    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:25.745423 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:25.745332    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:25.854163 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.854128    2562 generic.go:356] "Generic (PLEG): container finished" podID="f73652f1-9b2c-4387-b0a7-5d90a83e69c7" containerID="05d49e6ffe8d6032c71bbc0f051cb9f43fc28f96625d34267e36f80458cc11c9" exitCode=0
Jul 16 21:47:25.854325 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.854208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerDied","Data":"05d49e6ffe8d6032c71bbc0f051cb9f43fc28f96625d34267e36f80458cc11c9"}
Jul 16 21:47:25.856452 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.856429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc" event={"ID":"9b33f558-7844-420c-b60a-fbeef1574933","Type":"ContainerStarted","Data":"a75e4e78a942b466b6124ee01f67ddb149927e90d89a4df7d9b33f6947a7c512"}
Jul 16 21:47:25.860152 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.860132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" event={"ID":"145430b6-2caf-411f-a744-e590d9551eee","Type":"ContainerStarted","Data":"aa5fb885bc7a6f902a2c54b41c41e9a8fb6c6186d82d6e71b7ba9d9e06c71408"}
Jul 16 21:47:25.860429 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.860404    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:47:25.860502 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.860469    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:47:25.860502 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.860499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:47:25.876241 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.876222    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:47:25.876348 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.876313    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:47:25.914099 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:25.914031    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg" podStartSLOduration=7.919243214 podStartE2EDuration="27.914012819s" totalImagesPullingTime="19.994769605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:25.911228215 +0000 UTC m=+28.693126062" watchObservedRunningTime="2026-07-16 21:47:25.914012819 +0000 UTC m=+28.695910668"
Jul 16 21:47:26.744765 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:26.744730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:26.745380 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:26.744841    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:27.467109 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.466849    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8rlc" podStartSLOduration=5.057870975 podStartE2EDuration="29.466831838s" totalImagesPullingTime="24.408960863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:25.930926858 +0000 UTC m=+28.712824708" watchObservedRunningTime="2026-07-16 21:47:27.466831838 +0000 UTC m=+30.248729720"
Jul 16 21:47:27.467655 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.467633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h25ht"]
Jul 16 21:47:27.467747 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.467735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:27.467831 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:27.467814    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:27.470623 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.470594    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxb4t"]
Jul 16 21:47:27.470753 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.470703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:27.470815 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:27.470796    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:27.471376 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.471342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kbs6s"]
Jul 16 21:47:27.471487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.471457    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:27.471572 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:27.471560    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:27.627747 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.627702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:27.728217 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.728130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:27.866745 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.866714    2562 generic.go:356] "Generic (PLEG): container finished" podID="f73652f1-9b2c-4387-b0a7-5d90a83e69c7" containerID="add71d23c35e4c829a3c75183faafd0e7c1ccf1d97cdaa2ec129147fc458122d" exitCode=0
Jul 16 21:47:27.867129 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:27.866809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerDied","Data":"add71d23c35e4c829a3c75183faafd0e7c1ccf1d97cdaa2ec129147fc458122d"}
Jul 16 21:47:28.486207 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:28.486160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:28.566973 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:28.566932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:28.744950 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:28.744868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:28.745114 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:28.744976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:28.745114 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:28.744982    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:28.745114 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:28.745078    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:28.745114 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:28.744982    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:28.745261 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:28.745189    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:29.160911 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:29.160879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:29.874249 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:29.874163    2562 generic.go:356] "Generic (PLEG): container finished" podID="f73652f1-9b2c-4387-b0a7-5d90a83e69c7" containerID="be1c84699afd70aaae30729954f01ca7ad80176f24804837ea5b8239cf943aba" exitCode=0
Jul 16 21:47:29.874249 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:29.874205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerDied","Data":"be1c84699afd70aaae30729954f01ca7ad80176f24804837ea5b8239cf943aba"}
Jul 16 21:47:30.744756 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:30.744725    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:30.745157 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:30.744726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:30.745157 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:30.744843    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:30.745157 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:30.744903    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:30.745157 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:30.744726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:30.745157 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:30.744998    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:31.447897 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:31.447857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:31.448075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:31.447911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:31.448075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:31.447957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:31.448075 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448032    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:31.448075 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448065    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:31.448322 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448107    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:03.448087474 +0000 UTC m=+66.229985300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:31.448322 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448112    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:31.448322 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448124    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret podName:0bec7cf3-d1a3-4497-b829-932fbc703c4a nodeName:}" failed. No retries permitted until 2026-07-16 21:48:03.448117706 +0000 UTC m=+66.230015531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret") pod "global-pull-secret-syncer-h25ht" (UID: "0bec7cf3-d1a3-4497-b829-932fbc703c4a") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:31.448322 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448135    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:31.448322 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448148    2562 projected.go:196] Error preparing data for projected volume kube-api-access-ghqtm for pod openshift-network-diagnostics/network-check-target-sxb4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:31.448322 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:31.448191    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm podName:dee5aa63-6c77-4948-b731-1e5130327f8c nodeName:}" failed. No retries permitted until 2026-07-16 21:48:03.448175393 +0000 UTC m=+66.230073219 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghqtm" (UniqueName: "kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm") pod "network-check-target-sxb4t" (UID: "dee5aa63-6c77-4948-b731-1e5130327f8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:32.464346 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:32.464310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:32.559158 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:32.559113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:32.745815 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:32.745532    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:32.745962 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:32.745568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:32.745962 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:32.745865    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxb4t" podUID="dee5aa63-6c77-4948-b731-1e5130327f8c"
Jul 16 21:47:32.745962 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:32.745609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:32.746084 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:32.745959    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:47:32.746084 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:32.746038    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h25ht" podUID="0bec7cf3-d1a3-4497-b829-932fbc703c4a"
Jul 16 21:47:33.084180 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.084108    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-156.ec2.internal" event="NodeReady"
Jul 16 21:47:33.084336 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.084258    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:47:33.184951 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.184909    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ds5ls"]
Jul 16 21:47:33.187648 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.187618    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hqdk7"]
Jul 16 21:47:33.187770 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.187752    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.189638 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.189621    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.195099 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.195076    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:47:33.195511 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.195490    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-275zl\""
Jul 16 21:47:33.195727 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.195700    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:47:33.195806 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.195747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:47:33.196071 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.196043    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7z9z2\""
Jul 16 21:47:33.200905 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.196804    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:47:33.207341 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.207321    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:47:33.251044 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.250992    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ds5ls"]
Jul 16 21:47:33.253286 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.253264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hqdk7"]
Jul 16 21:47:33.365513 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.365418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbdd52d-ece0-471e-8645-b79df023e5ac-cert\") pod \"ingress-canary-ds5ls\" (UID: \"4dbdd52d-ece0-471e-8645-b79df023e5ac\") " pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.365513 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.365459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/607c5dc3-2461-4b05-a37a-413b4a681f38-config-volume\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.365513 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.365475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c5dc3-2461-4b05-a37a-413b4a681f38-tmp-dir\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.365776 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.365626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86htt\" (UniqueName: \"kubernetes.io/projected/4dbdd52d-ece0-471e-8645-b79df023e5ac-kube-api-access-86htt\") pod \"ingress-canary-ds5ls\" (UID: \"4dbdd52d-ece0-471e-8645-b79df023e5ac\") " pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.365776 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.365653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblmg\" (UniqueName: \"kubernetes.io/projected/607c5dc3-2461-4b05-a37a-413b4a681f38-kube-api-access-tblmg\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.365776 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.365684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.466240 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c5dc3-2461-4b05-a37a-413b4a681f38-tmp-dir\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86htt\" (UniqueName: \"kubernetes.io/projected/4dbdd52d-ece0-471e-8645-b79df023e5ac-kube-api-access-86htt\") pod \"ingress-canary-ds5ls\" (UID: \"4dbdd52d-ece0-471e-8645-b79df023e5ac\") " pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblmg\" (UniqueName: \"kubernetes.io/projected/607c5dc3-2461-4b05-a37a-413b4a681f38-kube-api-access-tblmg\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbdd52d-ece0-471e-8645-b79df023e5ac-cert\") pod \"ingress-canary-ds5ls\" (UID: \"4dbdd52d-ece0-471e-8645-b79df023e5ac\") " pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/607c5dc3-2461-4b05-a37a-413b4a681f38-config-volume\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:33.466573    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.466587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c5dc3-2461-4b05-a37a-413b4a681f38-tmp-dir\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.466726 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:33.466666    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:33.966643974 +0000 UTC m=+36.748541803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:33.467085 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.467063    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/607c5dc3-2461-4b05-a37a-413b4a681f38-config-volume\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.471064 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.471047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbdd52d-ece0-471e-8645-b79df023e5ac-cert\") pod \"ingress-canary-ds5ls\" (UID: \"4dbdd52d-ece0-471e-8645-b79df023e5ac\") " pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.479308 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.479285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86htt\" (UniqueName: \"kubernetes.io/projected/4dbdd52d-ece0-471e-8645-b79df023e5ac-kube-api-access-86htt\") pod \"ingress-canary-ds5ls\" (UID: \"4dbdd52d-ece0-471e-8645-b79df023e5ac\") " pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.483489 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.483469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblmg\" (UniqueName: \"kubernetes.io/projected/607c5dc3-2461-4b05-a37a-413b4a681f38-kube-api-access-tblmg\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.490184 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.490030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:33.500476 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.500452    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ds5ls"
Jul 16 21:47:33.969510 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.969477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:33.969701 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:33.969661    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:33.969776 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:33.969758    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:34.969735909 +0000 UTC m=+37.751633737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:33.973174 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:33.973148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:34.250452 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.250321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ds5ls"]
Jul 16 21:47:34.255295 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:34.255266    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbdd52d_ece0_471e_8645_b79df023e5ac.slice/crio-1f72f8ebd2a305db48230a00675e7507467901a017013a24c2dc75fb7bb9ad5a WatchSource:0}: Error finding container 1f72f8ebd2a305db48230a00675e7507467901a017013a24c2dc75fb7bb9ad5a: Status 404 returned error can't find the container with id 1f72f8ebd2a305db48230a00675e7507467901a017013a24c2dc75fb7bb9ad5a
Jul 16 21:47:34.536803 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.536672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:34.608637 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.608606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:34.745372 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.745328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:47:34.745607 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.745347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:47:34.745658 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.745357    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:47:34.748151 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.748127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:47:34.748151 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.748139    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:47:34.748426 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.748410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:47:34.749436 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.749419    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wc4rs\""
Jul 16 21:47:34.749538 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.749495    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gkkkc\""
Jul 16 21:47:34.749538 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.749502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:47:34.889610 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.889514    2562 generic.go:356] "Generic (PLEG): container finished" podID="f73652f1-9b2c-4387-b0a7-5d90a83e69c7" containerID="315d52b317ee07f2254e8e501699cbab2934c5d4671cbb7a18ea07cca62df2b8" exitCode=0
Jul 16 21:47:34.889610 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.889583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerDied","Data":"315d52b317ee07f2254e8e501699cbab2934c5d4671cbb7a18ea07cca62df2b8"}
Jul 16 21:47:34.891020 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.890992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ds5ls" event={"ID":"4dbdd52d-ece0-471e-8645-b79df023e5ac","Type":"ContainerStarted","Data":"1f72f8ebd2a305db48230a00675e7507467901a017013a24c2dc75fb7bb9ad5a"}
Jul 16 21:47:34.976862 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:34.976835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:34.977023 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:34.977000    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:34.977094 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:34.977082    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:36.977060877 +0000 UTC m=+39.758958706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:35.897504 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:35.897468    2562 generic.go:356] "Generic (PLEG): container finished" podID="f73652f1-9b2c-4387-b0a7-5d90a83e69c7" containerID="e8d3677ced7108c2fd9a3cc7bfc7682ab36db441670e1b668e0b1590436dc355" exitCode=0
Jul 16 21:47:35.897917 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:35.897539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerDied","Data":"e8d3677ced7108c2fd9a3cc7bfc7682ab36db441670e1b668e0b1590436dc355"}
Jul 16 21:47:36.903668 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:36.903638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kffsw" event={"ID":"f73652f1-9b2c-4387-b0a7-5d90a83e69c7","Type":"ContainerStarted","Data":"8533c076195a4743b430c36a3968217999e242c62972060356965c72a51fd577"}
Jul 16 21:47:36.905134 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:36.905115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ds5ls" event={"ID":"4dbdd52d-ece0-471e-8645-b79df023e5ac","Type":"ContainerStarted","Data":"d34110f7a9d726107fc9e5dc8cf3eafed0c2eb600b3969c01f45bc99460994a7"}
Jul 16 21:47:36.931436 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:36.931353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kffsw" podStartSLOduration=7.799117002 podStartE2EDuration="38.931333389s" totalImagesPullingTime="31.132216387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:36.931168405 +0000 UTC m=+39.713066255" watchObservedRunningTime="2026-07-16 21:47:36.931333389 +0000 UTC m=+39.713231215"
Jul 16 21:47:36.995443 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:36.995406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:36.995662 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:36.995599    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:36.995729 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:36.995678    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:40.995658129 +0000 UTC m=+43.777555973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:37.054561 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:37.054525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:37.128650 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:37.128608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:38.030333 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:38.030152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:38.135065 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:38.135026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:41.022865 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:41.022825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:41.023274 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:41.022977    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:41.023274 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:41.023041    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:49.023026828 +0000 UTC m=+51.804924654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:46.626210 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:46.626164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:46.698826 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:46.698795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:47.900333 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.900284    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ds5ls" podStartSLOduration=12.382124256 podStartE2EDuration="14.900266809s" totalImagesPullingTime="2.518142553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:37.934895328 +0000 UTC m=+40.716793175" watchObservedRunningTime="2026-07-16 21:47:47.900266809 +0000 UTC m=+50.682164669"
Jul 16 21:47:47.900727 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.900630    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"]
Jul 16 21:47:47.919749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.919721    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"]
Jul 16 21:47:47.919879 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.919773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:47.934158 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.934137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:47.968765 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.968731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/725c0637-9d96-49ec-92cb-3cbf2e416048-klusterlet-config\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:47.968884 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.968783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/725c0637-9d96-49ec-92cb-3cbf2e416048-tmp\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:47.968884 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.968800    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 21:47:47.968884 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.968807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c055adf-2e1e-4caf-86f1-c73b58e7a860-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d6c65b658-dl78l\" (UID: \"9c055adf-2e1e-4caf-86f1-c73b58e7a860\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:47.969025 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.968901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvfg\" (UniqueName: \"kubernetes.io/projected/725c0637-9d96-49ec-92cb-3cbf2e416048-kube-api-access-bmvfg\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:47.969025 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.968972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h62k\" (UniqueName: \"kubernetes.io/projected/9c055adf-2e1e-4caf-86f1-c73b58e7a860-kube-api-access-5h62k\") pod \"managed-serviceaccount-addon-agent-7d6c65b658-dl78l\" (UID: \"9c055adf-2e1e-4caf-86f1-c73b58e7a860\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:47.969749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.969580    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"]
Jul 16 21:47:47.969749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.969628    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 21:47:47.969749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.969724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wnj84\""
Jul 16 21:47:47.969749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.969734    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 16 21:47:47.969956 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.969723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 21:47:47.970448 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.970432    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 21:47:47.975602 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:47.975579    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"]
Jul 16 21:47:48.069948 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.069920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h62k\" (UniqueName: \"kubernetes.io/projected/9c055adf-2e1e-4caf-86f1-c73b58e7a860-kube-api-access-5h62k\") pod \"managed-serviceaccount-addon-agent-7d6c65b658-dl78l\" (UID: \"9c055adf-2e1e-4caf-86f1-c73b58e7a860\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:48.070102 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.070004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/725c0637-9d96-49ec-92cb-3cbf2e416048-klusterlet-config\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.070102 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.070049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/725c0637-9d96-49ec-92cb-3cbf2e416048-tmp\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.070102 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.070071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c055adf-2e1e-4caf-86f1-c73b58e7a860-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d6c65b658-dl78l\" (UID: \"9c055adf-2e1e-4caf-86f1-c73b58e7a860\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:48.070241 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.070220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvfg\" (UniqueName: \"kubernetes.io/projected/725c0637-9d96-49ec-92cb-3cbf2e416048-kube-api-access-bmvfg\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.070469 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.070449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/725c0637-9d96-49ec-92cb-3cbf2e416048-tmp\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.074234 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.074208    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/725c0637-9d96-49ec-92cb-3cbf2e416048-klusterlet-config\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.074337 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.074303    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c055adf-2e1e-4caf-86f1-c73b58e7a860-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d6c65b658-dl78l\" (UID: \"9c055adf-2e1e-4caf-86f1-c73b58e7a860\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:48.078872 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.078851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h62k\" (UniqueName: \"kubernetes.io/projected/9c055adf-2e1e-4caf-86f1-c73b58e7a860-kube-api-access-5h62k\") pod \"managed-serviceaccount-addon-agent-7d6c65b658-dl78l\" (UID: \"9c055adf-2e1e-4caf-86f1-c73b58e7a860\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:48.079037 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.079013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvfg\" (UniqueName: \"kubernetes.io/projected/725c0637-9d96-49ec-92cb-3cbf2e416048-kube-api-access-bmvfg\") pod \"klusterlet-addon-workmgr-74cf5bf866-895wz\" (UID: \"725c0637-9d96-49ec-92cb-3cbf2e416048\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.248667 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.248581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:47:48.255728 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.255711    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:48.416344 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.416319    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"]
Jul 16 21:47:48.419034 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:48.419004    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c055adf_2e1e_4caf_86f1_c73b58e7a860.slice/crio-0bf906d85fb9a9b45d90b0006c49ab7e42bded2e0449197d49be3d841f67cf89 WatchSource:0}: Error finding container 0bf906d85fb9a9b45d90b0006c49ab7e42bded2e0449197d49be3d841f67cf89: Status 404 returned error can't find the container with id 0bf906d85fb9a9b45d90b0006c49ab7e42bded2e0449197d49be3d841f67cf89
Jul 16 21:47:48.437083 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.437059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"]
Jul 16 21:47:48.439741 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:47:48.439717    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725c0637_9d96_49ec_92cb_3cbf2e416048.slice/crio-eb5f020f5ae498a9ebea0a35779d5139a0e113150e649075ea96f30bffcf687e WatchSource:0}: Error finding container eb5f020f5ae498a9ebea0a35779d5139a0e113150e649075ea96f30bffcf687e: Status 404 returned error can't find the container with id eb5f020f5ae498a9ebea0a35779d5139a0e113150e649075ea96f30bffcf687e
Jul 16 21:47:48.934486 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.934453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz" event={"ID":"725c0637-9d96-49ec-92cb-3cbf2e416048","Type":"ContainerStarted","Data":"eb5f020f5ae498a9ebea0a35779d5139a0e113150e649075ea96f30bffcf687e"}
Jul 16 21:47:48.935447 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:48.935426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" event={"ID":"9c055adf-2e1e-4caf-86f1-c73b58e7a860","Type":"ContainerStarted","Data":"0bf906d85fb9a9b45d90b0006c49ab7e42bded2e0449197d49be3d841f67cf89"}
Jul 16 21:47:49.075191 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:49.075159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:47:49.075368 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:49.075308    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:49.075368 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:47:49.075364    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.075348546 +0000 UTC m=+67.857246372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:51.945791 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:51.945750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" event={"ID":"9c055adf-2e1e-4caf-86f1-c73b58e7a860","Type":"ContainerStarted","Data":"8699137ca22b6d019d4e496df14f379e221cc5102b9cf89b30ca1533735e35b5"}
Jul 16 21:47:51.961106 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:51.961054    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" podStartSLOduration=1.862423041 podStartE2EDuration="4.961033648s" totalImagesPullingTime="3.098610607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:51.960609568 +0000 UTC m=+54.742507418" watchObservedRunningTime="2026-07-16 21:47:51.961033648 +0000 UTC m=+54.742931498"
Jul 16 21:47:52.950075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:52.950039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz" event={"ID":"725c0637-9d96-49ec-92cb-3cbf2e416048","Type":"ContainerStarted","Data":"8b257175ada904e2d67f23f55d0bb8320e282a064a628fa03af1e293ccd090a3"}
Jul 16 21:47:52.950481 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:52.950212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:52.951453 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:52.951433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:47:52.971524 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:52.971480    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz" podStartSLOduration=1.561851981 podStartE2EDuration="5.971467381s" totalImagesPullingTime="4.4096154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:52.970843973 +0000 UTC m=+55.752741820" watchObservedRunningTime="2026-07-16 21:47:52.971467381 +0000 UTC m=+55.753365228"
Jul 16 21:47:54.514057 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:54.514021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:54.613100 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:54.613068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:57.883066 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:57.883038    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cccgg"
Jul 16 21:47:58.181676 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:58.181646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:58.285368 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:58.285334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:47:58.370152 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:47:58.370118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:48:03.479706 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.479655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:48:03.479706 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.479716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:48:03.480199 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.479748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:48:03.484740 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.484721    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:48:03.484820 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.484773    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:48:03.484856 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.484833    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:48:03.490599 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:48:03.490580    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:48:03.490670 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:48:03.490644    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:07.490622951 +0000 UTC m=+130.272520777 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : secret "metrics-daemon-secret" not found
Jul 16 21:48:03.492825 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.492807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bec7cf3-d1a3-4497-b829-932fbc703c4a-original-pull-secret\") pod \"global-pull-secret-syncer-h25ht\" (UID: \"0bec7cf3-d1a3-4497-b829-932fbc703c4a\") " pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:48:03.493198 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.493186    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:48:03.503980 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.503958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghqtm\" (UniqueName: \"kubernetes.io/projected/dee5aa63-6c77-4948-b731-1e5130327f8c-kube-api-access-ghqtm\") pod \"network-check-target-sxb4t\" (UID: \"dee5aa63-6c77-4948-b731-1e5130327f8c\") " pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:48:03.558131 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.558103    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wc4rs\""
Jul 16 21:48:03.565959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.565938    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h25ht"
Jul 16 21:48:03.565959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.565948    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:48:03.730992 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.730969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h25ht"]
Jul 16 21:48:03.742667 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.742645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxb4t"]
Jul 16 21:48:03.745683 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:48:03.745660    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee5aa63_6c77_4948_b731_1e5130327f8c.slice/crio-4682f4cef8b9609c917e7d75541cc7ed3f14c4d338fdd351d9204fec0127af81 WatchSource:0}: Error finding container 4682f4cef8b9609c917e7d75541cc7ed3f14c4d338fdd351d9204fec0127af81: Status 404 returned error can't find the container with id 4682f4cef8b9609c917e7d75541cc7ed3f14c4d338fdd351d9204fec0127af81
Jul 16 21:48:03.976010 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.975965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h25ht" event={"ID":"0bec7cf3-d1a3-4497-b829-932fbc703c4a","Type":"ContainerStarted","Data":"0f83d6d790f593c8b9dc33d34e54b8d3bdbb586dbc9ee1b35ac39ee3e60f2bd3"}
Jul 16 21:48:03.977134 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:03.977110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxb4t" event={"ID":"dee5aa63-6c77-4948-b731-1e5130327f8c","Type":"ContainerStarted","Data":"4682f4cef8b9609c917e7d75541cc7ed3f14c4d338fdd351d9204fec0127af81"}
Jul 16 21:48:05.094891 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:05.094853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:48:05.095324 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:48:05.095018    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:05.095324 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:48:05.095072    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:37.095058476 +0000 UTC m=+99.876956302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:08.993232 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:08.993199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h25ht" event={"ID":"0bec7cf3-d1a3-4497-b829-932fbc703c4a","Type":"ContainerStarted","Data":"d2152dfb8d31fd9549d5aba925412a8a2dedeabb785e3d0622ffba71afd24a7b"}
Jul 16 21:48:08.994631 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:08.994608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxb4t" event={"ID":"dee5aa63-6c77-4948-b731-1e5130327f8c","Type":"ContainerStarted","Data":"603b9240cbccfea07c7f2ddd8e88eeb3bf604c55bbb8f097de0cfc7b05873ffd"}
Jul 16 21:48:08.994739 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:08.994727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:48:09.053026 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:09.052978    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h25ht" podStartSLOduration=66.457702592 podStartE2EDuration="1m11.052961633s" totalImagesPullingTime="4.595259041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:09.013858731 +0000 UTC m=+71.795756579" watchObservedRunningTime="2026-07-16 21:48:09.052961633 +0000 UTC m=+71.834859482"
Jul 16 21:48:20.341047 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:20.340913    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:48:23.468702 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:23.468665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:48:23.636215 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:23.636174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:48:37.122103 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:37.122064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:48:37.122456 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:48:37.122242    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:37.122456 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:48:37.122303    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:41.122286739 +0000 UTC m=+163.904184564 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:39.999303 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:39.999272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sxb4t"
Jul 16 21:48:40.029480 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:48:40.029424    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sxb4t" podStartSLOduration=97.45185576 podStartE2EDuration="1m42.029408456s" totalImagesPullingTime="4.577552696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:09.052803446 +0000 UTC m=+71.834701294" watchObservedRunningTime="2026-07-16 21:48:40.029408456 +0000 UTC m=+102.811306303"
Jul 16 21:49:07.529754 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:07.529715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:49:07.530194 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:49:07.529857    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:49:07.530194 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:49:07.529925    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:09.52991037 +0000 UTC m=+252.311808197 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : secret "metrics-daemon-secret" not found
Jul 16 21:49:36.206244 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:49:36.206196    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hqdk7" podUID="607c5dc3-2461-4b05-a37a-413b4a681f38"
Jul 16 21:49:36.210153 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:36.210133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:49:37.761653 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:49:37.761619    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:49:41.166439 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:41.166403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:49:41.166951 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:49:41.166569    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:41.166951 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:49:41.166655    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls podName:607c5dc3-2461-4b05-a37a-413b4a681f38 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:43.166634995 +0000 UTC m=+285.948532824 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls") pod "dns-default-hqdk7" (UID: "607c5dc3-2461-4b05-a37a-413b4a681f38") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:51.745712 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:51.745681    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:49:52.252239 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:52.252205    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c055adf-2e1e-4caf-86f1-c73b58e7a860" containerID="8699137ca22b6d019d4e496df14f379e221cc5102b9cf89b30ca1533735e35b5" exitCode=255
Jul 16 21:49:52.252423 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:52.252245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" event={"ID":"9c055adf-2e1e-4caf-86f1-c73b58e7a860","Type":"ContainerDied","Data":"8699137ca22b6d019d4e496df14f379e221cc5102b9cf89b30ca1533735e35b5"}
Jul 16 21:49:52.257731 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:52.257713    2562 scope.go:117] "RemoveContainer" containerID="8699137ca22b6d019d4e496df14f379e221cc5102b9cf89b30ca1533735e35b5"
Jul 16 21:49:52.951364 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:52.951323    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz" podUID="725c0637-9d96-49ec-92cb-3cbf2e416048" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.9:8000/readyz\": dial tcp 10.134.0.9:8000: connect: connection refused"
Jul 16 21:49:53.256775 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:53.256685    2562 generic.go:356] "Generic (PLEG): container finished" podID="725c0637-9d96-49ec-92cb-3cbf2e416048" containerID="8b257175ada904e2d67f23f55d0bb8320e282a064a628fa03af1e293ccd090a3" exitCode=1
Jul 16 21:49:53.256775 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:53.256762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz" event={"ID":"725c0637-9d96-49ec-92cb-3cbf2e416048","Type":"ContainerDied","Data":"8b257175ada904e2d67f23f55d0bb8320e282a064a628fa03af1e293ccd090a3"}
Jul 16 21:49:53.257145 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:53.257123    2562 scope.go:117] "RemoveContainer" containerID="8b257175ada904e2d67f23f55d0bb8320e282a064a628fa03af1e293ccd090a3"
Jul 16 21:49:53.258707 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:53.258681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" event={"ID":"9c055adf-2e1e-4caf-86f1-c73b58e7a860","Type":"ContainerStarted","Data":"f17127a7f08b3cd9a1a4dd0d24469e0777aa2572df3d37e55585eb026d69b45b"}
Jul 16 21:49:54.262540 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:54.262508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz" event={"ID":"725c0637-9d96-49ec-92cb-3cbf2e416048","Type":"ContainerStarted","Data":"0e9312c73d64a26fa6fb2f94909dee09bbc2c179fdeaf83cd1559358147ec9eb"}
Jul 16 21:49:54.262980 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:54.262811    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:49:54.263386 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:49:54.263368    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74cf5bf866-895wz"
Jul 16 21:50:21.735258 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:50:21.735221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:09.132163 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.132127    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"]
Jul 16 21:51:09.134254 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.134235    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-855975bf95-dpx9k"]
Jul 16 21:51:09.134378 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.134360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.136205 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.136185    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"]
Jul 16 21:51:09.136368 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.136352    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.137884 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.137865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"
Jul 16 21:51:09.138379 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.138357    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:51:09.138512 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.138412    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 16 21:51:09.138512 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.138503    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 16 21:51:09.138663 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.138648    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sjsqq\""
Jul 16 21:51:09.138663 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.138653    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 16 21:51:09.140022 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.140004    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:51:09.140767 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.140748    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:51:09.140867 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.140790    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:51:09.142092 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.142076    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5hjz7\""
Jul 16 21:51:09.142092 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.142077    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:51:09.143039 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.143021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j8r94\""
Jul 16 21:51:09.143039 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.143037    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 21:51:09.157596 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.157580    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:51:09.167675 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.167643    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-855975bf95-dpx9k"]
Jul 16 21:51:09.168619 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.168592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"]
Jul 16 21:51:09.184331 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.184305    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"]
Jul 16 21:51:09.269828 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-image-registry-private-configuration\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.269828 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269825    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-certificates\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e3df6b-688c-4e41-ae3a-147f7febee41-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-ca-trust-extracted\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-installation-pull-secrets\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e3df6b-688c-4e41-ae3a-147f7febee41-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.269996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2b68\" (UniqueName: \"kubernetes.io/projected/7cda6ed7-0b12-445e-8e15-4b378a623acf-kube-api-access-x2b68\") pod \"volume-data-source-validator-78cf4c6f6-9cldx\" (UID: \"7cda6ed7-0b12-445e-8e15-4b378a623acf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.270023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-trusted-ca\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.270043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9kk\" (UniqueName: \"kubernetes.io/projected/76e3df6b-688c-4e41-ae3a-147f7febee41-kube-api-access-kr9kk\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.270078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.270068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9rq\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-kube-api-access-nf9rq\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.270314 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.270110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.270314 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.270131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-bound-sa-token\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371185 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e3df6b-688c-4e41-ae3a-147f7febee41-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.371185 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-ca-trust-extracted\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-installation-pull-secrets\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e3df6b-688c-4e41-ae3a-147f7febee41-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b68\" (UniqueName: \"kubernetes.io/projected/7cda6ed7-0b12-445e-8e15-4b378a623acf-kube-api-access-x2b68\") pod \"volume-data-source-validator-78cf4c6f6-9cldx\" (UID: \"7cda6ed7-0b12-445e-8e15-4b378a623acf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-trusted-ca\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9kk\" (UniqueName: \"kubernetes.io/projected/76e3df6b-688c-4e41-ae3a-147f7febee41-kube-api-access-kr9kk\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9rq\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-kube-api-access-nf9rq\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371440 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-bound-sa-token\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-image-registry-private-configuration\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.371621    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-certificates\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.371636    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-855975bf95-dpx9k: secret "image-registry-tls" not found
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.371678    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls podName:5a77c88f-42c2-4639-b425-7a7c1a5fe1e3 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:09.871662198 +0000 UTC m=+252.653560024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls") pod "image-registry-855975bf95-dpx9k" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3") : secret "image-registry-tls" not found
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-ca-trust-extracted\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.371862 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.371850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e3df6b-688c-4e41-ae3a-147f7febee41-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.372340 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.372306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-certificates\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.372657 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.372637    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-trusted-ca\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.373831 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.373807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e3df6b-688c-4e41-ae3a-147f7febee41-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.374147 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.374131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-installation-pull-secrets\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.374292 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.374276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-image-registry-private-configuration\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.383643 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.383583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9kk\" (UniqueName: \"kubernetes.io/projected/76e3df6b-688c-4e41-ae3a-147f7febee41-kube-api-access-kr9kk\") pod \"kube-storage-version-migrator-operator-7658d9b749-7t9gj\" (UID: \"76e3df6b-688c-4e41-ae3a-147f7febee41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.384910 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.384891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-bound-sa-token\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.387021 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.386998    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9rq\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-kube-api-access-nf9rq\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.389886 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.389862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2b68\" (UniqueName: \"kubernetes.io/projected/7cda6ed7-0b12-445e-8e15-4b378a623acf-kube-api-access-x2b68\") pod \"volume-data-source-validator-78cf4c6f6-9cldx\" (UID: \"7cda6ed7-0b12-445e-8e15-4b378a623acf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"
Jul 16 21:51:09.445570 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.445513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"
Jul 16 21:51:09.456165 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.456138    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"
Jul 16 21:51:09.459235 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.459209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:09.560873 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.560848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:09.573082 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.573052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:51:09.573233 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.573215    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:51:09.573346 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.573286    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs podName:10a2d502-339d-4f01-83ee-966860bc7a48 nodeName:}" failed. No retries permitted until 2026-07-16 21:53:11.573267858 +0000 UTC m=+374.355165684 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs") pod "network-metrics-daemon-kbs6s" (UID: "10a2d502-339d-4f01-83ee-966860bc7a48") : secret "metrics-daemon-secret" not found
Jul 16 21:51:09.586181 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.586154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj"]
Jul 16 21:51:09.590096 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:09.590072    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e3df6b_688c_4e41_ae3a_147f7febee41.slice/crio-fa7a240a7e04563bb9d4971be8cfd7dae9c9084d479c2e2eed24f75133484c82 WatchSource:0}: Error finding container fa7a240a7e04563bb9d4971be8cfd7dae9c9084d479c2e2eed24f75133484c82: Status 404 returned error can't find the container with id fa7a240a7e04563bb9d4971be8cfd7dae9c9084d479c2e2eed24f75133484c82
Jul 16 21:51:09.600895 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.600871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx"]
Jul 16 21:51:09.603638 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:09.603616    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cda6ed7_0b12_445e_8e15_4b378a623acf.slice/crio-09f1e977cf622f57fa3860799bbf4a2b7d4b1748821b3503f229159a8ed883fb WatchSource:0}: Error finding container 09f1e977cf622f57fa3860799bbf4a2b7d4b1748821b3503f229159a8ed883fb: Status 404 returned error can't find the container with id 09f1e977cf622f57fa3860799bbf4a2b7d4b1748821b3503f229159a8ed883fb
Jul 16 21:51:09.875565 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:09.875515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:09.875733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.875661    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:09.875733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.875678    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-855975bf95-dpx9k: secret "image-registry-tls" not found
Jul 16 21:51:09.875733 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:09.875733    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls podName:5a77c88f-42c2-4639-b425-7a7c1a5fe1e3 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:10.875718584 +0000 UTC m=+253.657616410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls") pod "image-registry-855975bf95-dpx9k" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3") : secret "image-registry-tls" not found
Jul 16 21:51:10.481358 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:10.481305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj" event={"ID":"76e3df6b-688c-4e41-ae3a-147f7febee41","Type":"ContainerStarted","Data":"fa7a240a7e04563bb9d4971be8cfd7dae9c9084d479c2e2eed24f75133484c82"}
Jul 16 21:51:10.483303 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:10.483268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx" event={"ID":"7cda6ed7-0b12-445e-8e15-4b378a623acf","Type":"ContainerStarted","Data":"09f1e977cf622f57fa3860799bbf4a2b7d4b1748821b3503f229159a8ed883fb"}
Jul 16 21:51:10.884158 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:10.884128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:10.884334 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:10.884311    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:10.884396 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:10.884339    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-855975bf95-dpx9k: secret "image-registry-tls" not found
Jul 16 21:51:10.884451 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:10.884420    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls podName:5a77c88f-42c2-4639-b425-7a7c1a5fe1e3 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:12.884399076 +0000 UTC m=+255.666296925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls") pod "image-registry-855975bf95-dpx9k" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3") : secret "image-registry-tls" not found
Jul 16 21:51:11.488701 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:11.488657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj" event={"ID":"76e3df6b-688c-4e41-ae3a-147f7febee41","Type":"ContainerStarted","Data":"db843f6e41bdeb4c00da46545f22ffec287a51c17b34682ae86839b85338f7f0"}
Jul 16 21:51:11.490475 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:11.490432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx" event={"ID":"7cda6ed7-0b12-445e-8e15-4b378a623acf","Type":"ContainerStarted","Data":"146a8265f5df8a8304951df39da20e8ec65be0fd88458aab610c15bd59a1373d"}
Jul 16 21:51:11.507627 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:11.507579    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj" podStartSLOduration=0.697633979 podStartE2EDuration="2.507563517s" totalImagesPullingTime="1.809929538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:11.506748911 +0000 UTC m=+254.288646759" watchObservedRunningTime="2026-07-16 21:51:11.507563517 +0000 UTC m=+254.289461355"
Jul 16 21:51:11.530054 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:11.529998    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9cldx" podStartSLOduration=1.285487129 podStartE2EDuration="2.529977382s" totalImagesPullingTime="1.244490253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:11.528748061 +0000 UTC m=+254.310645921" watchObservedRunningTime="2026-07-16 21:51:11.529977382 +0000 UTC m=+254.311875232"
Jul 16 21:51:11.633564 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:11.633518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:11.738531 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:11.738450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:12.303488 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.303454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:12.347699 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.347658    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"]
Jul 16 21:51:12.350853 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.350832    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"
Jul 16 21:51:12.355243 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.355226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 16 21:51:12.355447 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.355426    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 16 21:51:12.355676 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.355663    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-z2fmr\""
Jul 16 21:51:12.361327 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.361306    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"]
Jul 16 21:51:12.396702 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.396675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v9z\" (UniqueName: \"kubernetes.io/projected/d5ded2d9-965e-4a28-a63f-7fb80d6a0960-kube-api-access-92v9z\") pod \"migrator-7cc9c7ffd7-sssnm\" (UID: \"d5ded2d9-965e-4a28-a63f-7fb80d6a0960\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"
Jul 16 21:51:12.398800 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.398761    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:12.497297 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.497269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v9z\" (UniqueName: \"kubernetes.io/projected/d5ded2d9-965e-4a28-a63f-7fb80d6a0960-kube-api-access-92v9z\") pod \"migrator-7cc9c7ffd7-sssnm\" (UID: \"d5ded2d9-965e-4a28-a63f-7fb80d6a0960\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"
Jul 16 21:51:12.498493 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.498466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:12.507932 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.507913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v9z\" (UniqueName: \"kubernetes.io/projected/d5ded2d9-965e-4a28-a63f-7fb80d6a0960-kube-api-access-92v9z\") pod \"migrator-7cc9c7ffd7-sssnm\" (UID: \"d5ded2d9-965e-4a28-a63f-7fb80d6a0960\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"
Jul 16 21:51:12.659712 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.659685    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"
Jul 16 21:51:12.791387 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.791364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm"]
Jul 16 21:51:12.793710 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:12.793685    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ded2d9_965e_4a28_a63f_7fb80d6a0960.slice/crio-bab34f00009b497b4068d201719bb43210ded502de80ab824c17a5701fed31cb WatchSource:0}: Error finding container bab34f00009b497b4068d201719bb43210ded502de80ab824c17a5701fed31cb: Status 404 returned error can't find the container with id bab34f00009b497b4068d201719bb43210ded502de80ab824c17a5701fed31cb
Jul 16 21:51:12.899093 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:12.899061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:12.899281 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:12.899220    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:12.899281 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:12.899239    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-855975bf95-dpx9k: secret "image-registry-tls" not found
Jul 16 21:51:12.899396 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:12.899301    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls podName:5a77c88f-42c2-4639-b425-7a7c1a5fe1e3 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:16.899282271 +0000 UTC m=+259.681180116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls") pod "image-registry-855975bf95-dpx9k" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3") : secret "image-registry-tls" not found
Jul 16 21:51:13.497369 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.497336    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm" event={"ID":"d5ded2d9-965e-4a28-a63f-7fb80d6a0960","Type":"ContainerStarted","Data":"bab34f00009b497b4068d201719bb43210ded502de80ab824c17a5701fed31cb"}
Jul 16 21:51:13.872371 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.872283    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f2r7n"]
Jul 16 21:51:13.875798 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.875773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:13.878476 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.878441    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:51:13.878602 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.878537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:51:13.879414 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.879385    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:51:13.879414 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.879384    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:51:13.879606 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.879384    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvlkz\""
Jul 16 21:51:13.897787 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.897759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2r7n"]
Jul 16 21:51:13.905852 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.905810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qltn\" (UniqueName: \"kubernetes.io/projected/76683669-4321-4952-9731-007a95935967-kube-api-access-5qltn\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:13.905978 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.905937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:13.906049 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.905996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76683669-4321-4952-9731-007a95935967-data-volume\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:13.906049 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.906025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76683669-4321-4952-9731-007a95935967-crio-socket\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:13.906139 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.906054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:13.920278 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:13.920253    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.006610 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76683669-4321-4952-9731-007a95935967-data-volume\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.006786 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76683669-4321-4952-9731-007a95935967-crio-socket\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.006786 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.006786 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qltn\" (UniqueName: \"kubernetes.io/projected/76683669-4321-4952-9731-007a95935967-kube-api-access-5qltn\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.006786 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006769    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76683669-4321-4952-9731-007a95935967-crio-socket\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.006976 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.006830    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.006976 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.006976 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.006900    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:14.506878225 +0000 UTC m=+257.288776055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.006976 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.006953    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.006976 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.006967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76683669-4321-4952-9731-007a95935967-data-volume\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.007204 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.007004    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:14.506987434 +0000 UTC m=+257.288885260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.016114 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.016079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qltn\" (UniqueName: \"kubernetes.io/projected/76683669-4321-4952-9731-007a95935967-kube-api-access-5qltn\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.056923 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.056890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.166260 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.166225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.259381 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.259357    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.502321 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.502228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm" event={"ID":"d5ded2d9-965e-4a28-a63f-7fb80d6a0960","Type":"ContainerStarted","Data":"507fadb3f3ba3617ad970e8a757499ba01f2f6122a3feda7d81c606a4fa85bf8"}
Jul 16 21:51:14.502321 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.502276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm" event={"ID":"d5ded2d9-965e-4a28-a63f-7fb80d6a0960","Type":"ContainerStarted","Data":"396a3a7c9276433aa08f1475f165cf7192849daeba8028e5072f391b4415a6f4"}
Jul 16 21:51:14.511212 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.511189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.511354 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.511252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:14.511354 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.511326    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.511461 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.511354    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.511461 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.511374    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:15.511361655 +0000 UTC m=+258.293259481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.511461 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:14.511414    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:15.511395954 +0000 UTC m=+258.293293793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.675352 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.675306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.767912 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.767841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.957124 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:14.957090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:15.052196 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:15.052098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:15.516729 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:15.516700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:15.517117 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:15.516753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:15.517117 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:15.516828    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:15.517117 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:15.516836    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:15.517117 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:15.516885    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:17.516870358 +0000 UTC m=+260.298768185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:15.517117 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:15.516899    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:17.516892406 +0000 UTC m=+260.298790231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:16.473637 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:16.473606    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-98t4h_4e344137-20c3-47a9-97c0-a800bf47e475/dns-node-resolver/0.log"
Jul 16 21:51:16.502902 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:16.502875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.601187 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:16.601155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.925659 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:16.925623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:16.925840 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:16.925788    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:16.925840 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:16.925808    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-855975bf95-dpx9k: secret "image-registry-tls" not found
Jul 16 21:51:16.925951 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:16.925878    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls podName:5a77c88f-42c2-4639-b425-7a7c1a5fe1e3 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:24.925856595 +0000 UTC m=+267.707754435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls") pod "image-registry-855975bf95-dpx9k" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3") : secret "image-registry-tls" not found
Jul 16 21:51:17.451890 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:17.451865    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6q4k4_db43cfec-8a7b-4fc6-8370-c6f8dae02a69/node-ca/0.log"
Jul 16 21:51:17.529650 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:17.529612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:17.529807 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:17.529713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:17.529807 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:17.529732    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:17.529807 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:17.529788    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.529773178 +0000 UTC m=+264.311671004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:17.529907 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:17.529829    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:17.529907 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:17.529876    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.529864227 +0000 UTC m=+264.311762054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:17.684719 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:17.684685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:18.446665 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:18.446639    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ds5ls_4dbdd52d-ece0-471e-8645-b79df023e5ac/serve-healthcheck-canary/0.log"
Jul 16 21:51:18.849821 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:18.849735    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sssnm_d5ded2d9-965e-4a28-a63f-7fb80d6a0960/migrator/0.log"
Jul 16 21:51:19.046396 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:19.046364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sssnm_d5ded2d9-965e-4a28-a63f-7fb80d6a0960/graceful-termination/0.log"
Jul 16 21:51:19.257227 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:19.257192    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-7t9gj_76e3df6b-688c-4e41-ae3a-147f7febee41/kube-storage-version-migrator-operator/0.log"
Jul 16 21:51:20.322008 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:20.321974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:20.439922 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:20.439892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:20.602460 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:20.602390    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:20.743794 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:20.743758    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:20.853901 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:20.853816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:21.409725 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:21.409693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:21.557995 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:21.557959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:21.558133 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:21.558009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:21.558133 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:21.558109    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:21.558133 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:21.558122    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:21.558254 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:21.558185    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:29.558168894 +0000 UTC m=+272.340066720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:21.558254 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:21.558200    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:29.558193209 +0000 UTC m=+272.340091034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:22.065715 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:22.065682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:22.415246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:22.415215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:22.677285 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:22.677203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:22.826779 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:22.826747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:23.035795 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:23.035711    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:24.076533 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.076503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:24.197061 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.197034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:24.408940 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.408914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:24.521806 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.521774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:24.639651 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.639613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:24.981064 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.981035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:24.983508 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.983481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"image-registry-855975bf95-dpx9k\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") " pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:24.996525 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:24.996492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:25.050779 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.050734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:25.147459 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.147429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:25.203339 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.203293    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sssnm" podStartSLOduration=11.790919462 podStartE2EDuration="13.203273413s" totalImagesPullingTime="1.412353951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:14.540673889 +0000 UTC m=+257.322571737" watchObservedRunningTime="2026-07-16 21:51:25.203273413 +0000 UTC m=+267.985171271"
Jul 16 21:51:25.204056 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.204038    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-855975bf95-dpx9k"]
Jul 16 21:51:25.206731 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:25.206705    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a77c88f_42c2_4639_b425_7a7c1a5fe1e3.slice/crio-0c9589f350315ce23262fe960258a91fd9fae76356a8d3acba6f86bd5d393190 WatchSource:0}: Error finding container 0c9589f350315ce23262fe960258a91fd9fae76356a8d3acba6f86bd5d393190: Status 404 returned error can't find the container with id 0c9589f350315ce23262fe960258a91fd9fae76356a8d3acba6f86bd5d393190
Jul 16 21:51:25.535928 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.535851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-855975bf95-dpx9k" event={"ID":"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3","Type":"ContainerStarted","Data":"2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0"}
Jul 16 21:51:25.535928 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.535905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-855975bf95-dpx9k" event={"ID":"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3","Type":"ContainerStarted","Data":"0c9589f350315ce23262fe960258a91fd9fae76356a8d3acba6f86bd5d393190"}
Jul 16 21:51:25.536098 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.536050    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:25.571190 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:25.571131    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-855975bf95-dpx9k" podStartSLOduration=16.571107889 podStartE2EDuration="16.571107889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:25.569439232 +0000 UTC m=+268.351337080" watchObservedRunningTime="2026-07-16 21:51:25.571107889 +0000 UTC m=+268.353005739"
Jul 16 21:51:26.250685 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:26.250652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.437523 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:26.437482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.629242 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:26.629147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.829022 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:26.828975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.936021 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:26.935990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.034098 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:27.034060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:28.007651 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:28.007621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:28.510537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:28.510504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:28.632472 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:28.632438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:28.725106 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:28.725073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:29.015185 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:29.015157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:29.084325 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:29.084293    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:29.616010 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:29.615973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:29.616184 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:29.616065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:29.616184 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:29.616177    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:29.616258 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:29.616250    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls podName:76683669-4321-4952-9731-007a95935967 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:45.616230189 +0000 UTC m=+288.398128019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2r7n" (UID: "76683669-4321-4952-9731-007a95935967") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:29.616682 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:29.616662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76683669-4321-4952-9731-007a95935967-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:29.903786 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:29.903755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:30.217308 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:30.217204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:30.424133 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:30.424097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:30.605200 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:30.605100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:30.812119 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:30.812079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:31.025886 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:31.025836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:35.048332 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:35.048294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:36.927810 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:36.927768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:37.060305 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:37.060268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:37.149313 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:37.149277    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:37.687537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:37.687503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:37.788436 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:37.788406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:38.877506 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:38.877470    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-855975bf95-dpx9k"]
Jul 16 21:51:39.075353 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:39.075318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:39.186692 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:39.186663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:39.210958 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:39.210921    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hqdk7" podUID="607c5dc3-2461-4b05-a37a-413b4a681f38"
Jul 16 21:51:39.575837 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:39.575753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:51:41.110265 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:41.110229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:41.211319 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:41.211278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.214676 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.214635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:51:43.217232 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.217202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/607c5dc3-2461-4b05-a37a-413b4a681f38-metrics-tls\") pod \"dns-default-hqdk7\" (UID: \"607c5dc3-2461-4b05-a37a-413b4a681f38\") " pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:51:43.271371 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.271336    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.372498 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.372466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.480320 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.480247    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-275zl\""
Jul 16 21:51:43.487618 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.487594    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:51:43.628761 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:43.628729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hqdk7"]
Jul 16 21:51:43.632041 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:43.632011    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607c5dc3_2461_4b05_a37a_413b4a681f38.slice/crio-dea43333a4a4c3209c98c5be271dba8a938c42498290add4d352c0be2cc8af8e WatchSource:0}: Error finding container dea43333a4a4c3209c98c5be271dba8a938c42498290add4d352c0be2cc8af8e: Status 404 returned error can't find the container with id dea43333a4a4c3209c98c5be271dba8a938c42498290add4d352c0be2cc8af8e
Jul 16 21:51:44.151713 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:44.151674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:44.283093 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:44.283060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:44.416811 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:44.416728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:44.519487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:44.519448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:44.595735 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:44.595666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hqdk7" event={"ID":"607c5dc3-2461-4b05-a37a-413b4a681f38","Type":"ContainerStarted","Data":"dea43333a4a4c3209c98c5be271dba8a938c42498290add4d352c0be2cc8af8e"}
Jul 16 21:51:45.600405 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.600365    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hqdk7" event={"ID":"607c5dc3-2461-4b05-a37a-413b4a681f38","Type":"ContainerStarted","Data":"24435a5f97fefd8bf098e895c153c3274cb96841da0285c841dcc73c0304d217"}
Jul 16 21:51:45.600888 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.600427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hqdk7" event={"ID":"607c5dc3-2461-4b05-a37a-413b4a681f38","Type":"ContainerStarted","Data":"c2788b3677cf678a32a50ab116f46aa441706bd1f92fdf681489732ed5daae6b"}
Jul 16 21:51:45.600888 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.600491    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:51:45.625665 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.625619    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hqdk7" podStartSLOduration=251.263108923 podStartE2EDuration="4m12.625603456s" totalImagesPullingTime="1.362494533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:45.624311205 +0000 UTC m=+288.406209057" watchObservedRunningTime="2026-07-16 21:51:45.625603456 +0000 UTC m=+288.407501304"
Jul 16 21:51:45.635531 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.635502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:45.638093 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.638066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76683669-4321-4952-9731-007a95935967-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2r7n\" (UID: \"76683669-4321-4952-9731-007a95935967\") " pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:45.687186 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.687154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2r7n"
Jul 16 21:51:45.822917 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:45.822881    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2r7n"]
Jul 16 21:51:45.825855 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:45.825828    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76683669_4321_4952_9731_007a95935967.slice/crio-b2622baeb10ada530e146883b8172b6ad73708ceeef573267e75ff4f9978200d WatchSource:0}: Error finding container b2622baeb10ada530e146883b8172b6ad73708ceeef573267e75ff4f9978200d: Status 404 returned error can't find the container with id b2622baeb10ada530e146883b8172b6ad73708ceeef573267e75ff4f9978200d
Jul 16 21:51:46.107940 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:46.107909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:46.202302 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:46.202268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:46.606568 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:46.606512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2r7n" event={"ID":"76683669-4321-4952-9731-007a95935967","Type":"ContainerStarted","Data":"b2622baeb10ada530e146883b8172b6ad73708ceeef573267e75ff4f9978200d"}
Jul 16 21:51:48.399986 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.399947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:48.434967 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.434936    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"]
Jul 16 21:51:48.448864 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.448401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.449913 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.449861    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"]
Jul 16 21:51:48.451244 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.451219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:51:48.451244 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.451235    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 21:51:48.451511 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.451496    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 21:51:48.451599 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.451534    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:51:48.451599 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.451579    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nlmnm\""
Jul 16 21:51:48.452595 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.452572    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:51:48.507783 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.507723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:48.559475 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.559443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.559660 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.559490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkm7\" (UniqueName: \"kubernetes.io/projected/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-kube-api-access-bvkm7\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.559660 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.559520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.559660 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.559588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.660896 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.660816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkm7\" (UniqueName: \"kubernetes.io/projected/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-kube-api-access-bvkm7\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.660896 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.660862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.661124 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.660922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.661124 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.660984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.661124 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:48.661069    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 16 21:51:48.661321 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:48.661138    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-tls podName:6ea83b32-dcb9-4d65-a387-8ea4c6f878cf nodeName:}" failed. No retries permitted until 2026-07-16 21:51:49.161117565 +0000 UTC m=+291.943015395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-zv6dz" (UID: "6ea83b32-dcb9-4d65-a387-8ea4c6f878cf") : secret "prometheus-operator-tls" not found
Jul 16 21:51:48.661779 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.661747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.663686 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.663667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.672716 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.672688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkm7\" (UniqueName: \"kubernetes.io/projected/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-kube-api-access-bvkm7\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:48.884448 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.884208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:51:48.885891 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:48.885848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:49.165766 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:49.165731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:49.168735 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:49.168693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ea83b32-dcb9-4d65-a387-8ea4c6f878cf-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-zv6dz\" (UID: \"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:49.359681 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:49.359649    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"
Jul 16 21:51:49.991288 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:49.991238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:50.098409 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:50.098371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:50.751353 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:50.751303    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz"]
Jul 16 21:51:50.753925 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:50.753895    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea83b32_dcb9_4d65_a387_8ea4c6f878cf.slice/crio-3c591b7d83027488b87f2d23bedd6583996efb34dbf94f776039ace14c4c9f7d WatchSource:0}: Error finding container 3c591b7d83027488b87f2d23bedd6583996efb34dbf94f776039ace14c4c9f7d: Status 404 returned error can't find the container with id 3c591b7d83027488b87f2d23bedd6583996efb34dbf94f776039ace14c4c9f7d
Jul 16 21:51:51.626834 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:51.626796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz" event={"ID":"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf","Type":"ContainerStarted","Data":"3c591b7d83027488b87f2d23bedd6583996efb34dbf94f776039ace14c4c9f7d"}
Jul 16 21:51:51.630579 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:51.630502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2r7n" event={"ID":"76683669-4321-4952-9731-007a95935967","Type":"ContainerStarted","Data":"a6a126633fa9fd355201825c7f16fd13ee8a0a1d2b6e8792d71e2bc3724dc62e"}
Jul 16 21:51:52.273112 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.273061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.375026 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.374925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.477013 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.476977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.567428 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.567393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.636633 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.636533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz" event={"ID":"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf","Type":"ContainerStarted","Data":"4fad67ee3e4fb3b663b403e21d91406f4b41e9c81c784025ee38d02dcc14fb3d"}
Jul 16 21:51:52.637088 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.636656    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz" event={"ID":"6ea83b32-dcb9-4d65-a387-8ea4c6f878cf","Type":"ContainerStarted","Data":"06e621270c6a87863d58314892741ffd9f579b6131fcb1aa3d9b108ce81c9e81"}
Jul 16 21:51:52.676191 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:52.676132    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-zv6dz" podStartSLOduration=3.382680768 podStartE2EDuration="4.676116172s" totalImagesPullingTime="1.293435404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:52.675736334 +0000 UTC m=+295.457634186" watchObservedRunningTime="2026-07-16 21:51:52.676116172 +0000 UTC m=+295.458014024"
Jul 16 21:51:53.642781 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:53.642751    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c055adf-2e1e-4caf-86f1-c73b58e7a860" containerID="f17127a7f08b3cd9a1a4dd0d24469e0777aa2572df3d37e55585eb026d69b45b" exitCode=255
Jul 16 21:51:53.643241 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:53.642821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" event={"ID":"9c055adf-2e1e-4caf-86f1-c73b58e7a860","Type":"ContainerDied","Data":"f17127a7f08b3cd9a1a4dd0d24469e0777aa2572df3d37e55585eb026d69b45b"}
Jul 16 21:51:53.643241 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:53.642872    2562 scope.go:117] "RemoveContainer" containerID="8699137ca22b6d019d4e496df14f379e221cc5102b9cf89b30ca1533735e35b5"
Jul 16 21:51:53.650841 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:53.650820    2562 scope.go:117] "RemoveContainer" containerID="f17127a7f08b3cd9a1a4dd0d24469e0777aa2572df3d37e55585eb026d69b45b"
Jul 16 21:51:53.651041 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:53.651016    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7d6c65b658-dl78l_open-cluster-management-agent-addon(9c055adf-2e1e-4caf-86f1-c73b58e7a860)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" podUID="9c055adf-2e1e-4caf-86f1-c73b58e7a860"
Jul 16 21:51:54.181065 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.181033    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.271890 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.271862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.372728 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.372699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.616493 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.616387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.649180 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.649148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2r7n" event={"ID":"76683669-4321-4952-9731-007a95935967","Type":"ContainerStarted","Data":"5d2ca1d45a6323b9cfb18c3e0f89f7d2f548ed293b20697b80e493579618f00e"}
Jul 16 21:51:54.747076 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:54.747040    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kbs6s" podUID="10a2d502-339d-4f01-83ee-966860bc7a48"
Jul 16 21:51:54.829142 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.829109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.929330 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.929301    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9pdpx"]
Jul 16 21:51:54.932479 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.932449    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:54.935735 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.935715    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:51:54.935735 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.935730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:51:54.936230 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.936210    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l4dh8\""
Jul 16 21:51:54.936345 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.936210    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:51:54.942702 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:54.942676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.021244 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-sys\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021244 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-tls\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-textfile\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfe233-77be-496f-9683-9b61adea1438-metrics-client-ca\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021427    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr794\" (UniqueName: \"kubernetes.io/projected/b8dfe233-77be-496f-9683-9b61adea1438-kube-api-access-pr794\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-wtmp\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.021487 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.021488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-root\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.090647 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.090612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.122783 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.122746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr794\" (UniqueName: \"kubernetes.io/projected/b8dfe233-77be-496f-9683-9b61adea1438-kube-api-access-pr794\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.122959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.122793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-wtmp\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.122959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.122824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-root\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.122959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.122888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-sys\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.122959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.122913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-tls\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.122959 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.122938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123220 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-textfile\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123220 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-wtmp\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123220 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfe233-77be-496f-9683-9b61adea1438-metrics-client-ca\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123220 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-root\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123220 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123220 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:55.123192    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 21:51:55.123501 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:55.123252    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-tls podName:b8dfe233-77be-496f-9683-9b61adea1438 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:55.623232532 +0000 UTC m=+298.405130360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-tls") pod "node-exporter-9pdpx" (UID: "b8dfe233-77be-496f-9683-9b61adea1438") : secret "node-exporter-tls" not found
Jul 16 21:51:55.123501 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfe233-77be-496f-9683-9b61adea1438-sys\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123759 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-textfile\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.123915 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfe233-77be-496f-9683-9b61adea1438-metrics-client-ca\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.124028 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.123987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.125893 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.125874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.133352 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.133251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr794\" (UniqueName: \"kubernetes.io/projected/b8dfe233-77be-496f-9683-9b61adea1438-kube-api-access-pr794\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.318953 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.318868    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.419316 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.419268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.593336 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.593234    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.609278 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.609250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hqdk7"
Jul 16 21:51:55.628663 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.628636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-tls\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.631690 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.631661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfe233-77be-496f-9683-9b61adea1438-node-exporter-tls\") pod \"node-exporter-9pdpx\" (UID: \"b8dfe233-77be-496f-9683-9b61adea1438\") " pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.767729 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.767692    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.858288 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.858206    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pdpx"
Jul 16 21:51:55.870854 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.870736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.874672 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:51:55.874632    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dfe233_77be_496f_9683_9b61adea1438.slice/crio-e0793a943ed2ba00b2849414988d41df23c3b3cbe2a66887ec6eda5f20ae3953 WatchSource:0}: Error finding container e0793a943ed2ba00b2849414988d41df23c3b3cbe2a66887ec6eda5f20ae3953: Status 404 returned error can't find the container with id e0793a943ed2ba00b2849414988d41df23c3b3cbe2a66887ec6eda5f20ae3953
Jul 16 21:51:55.948030 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:55.947991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:56.067627 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:56.067597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:56.660193 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:56.660152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pdpx" event={"ID":"b8dfe233-77be-496f-9683-9b61adea1438","Type":"ContainerStarted","Data":"e0793a943ed2ba00b2849414988d41df23c3b3cbe2a66887ec6eda5f20ae3953"}
Jul 16 21:51:57.770064 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:57.769879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:57.870770 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:57.870736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:57.965108 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:57.965085    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:51:58.131991 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.131940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:58.248914 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.248884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l"
Jul 16 21:51:58.255914 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.255894    2562 scope.go:117] "RemoveContainer" containerID="f17127a7f08b3cd9a1a4dd0d24469e0777aa2572df3d37e55585eb026d69b45b"
Jul 16 21:51:58.256102 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:51:58.256081    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7d6c65b658-dl78l_open-cluster-management-agent-addon(9c055adf-2e1e-4caf-86f1-c73b58e7a860)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" podUID="9c055adf-2e1e-4caf-86f1-c73b58e7a860"
Jul 16 21:51:58.457537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.457498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:58.550456 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.550428    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:58.669904 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.669879    2562 generic.go:356] "Generic (PLEG): container finished" podID="b8dfe233-77be-496f-9683-9b61adea1438" containerID="cf755b6eacdf91a7a1de6dcfdffc236340babb20333c013d61e951d8cba9ec7f" exitCode=0
Jul 16 21:51:58.670035 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.669966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pdpx" event={"ID":"b8dfe233-77be-496f-9683-9b61adea1438","Type":"ContainerDied","Data":"cf755b6eacdf91a7a1de6dcfdffc236340babb20333c013d61e951d8cba9ec7f"}
Jul 16 21:51:58.672003 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.671983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2r7n" event={"ID":"76683669-4321-4952-9731-007a95935967","Type":"ContainerStarted","Data":"c8d689b7db8b1b1b0a3176e06ea1bdd54959ca59efa46d5a51b03c2b1eb6307d"}
Jul 16 21:51:58.673058 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:58.673037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.011234 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.011155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.090365 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.090331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.664089 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.664051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.677193 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.677159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pdpx" event={"ID":"b8dfe233-77be-496f-9683-9b61adea1438","Type":"ContainerStarted","Data":"fc1fd42f017d8ea15fd9954cb94891b72928ee2f7785836cbc9a78d141375186"}
Jul 16 21:51:59.677351 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.677243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pdpx" event={"ID":"b8dfe233-77be-496f-9683-9b61adea1438","Type":"ContainerStarted","Data":"fbb0b0d2af361e54a1bd1576dfab4a18add9073aa238155be23dd7fb6c07fbb1"}
Jul 16 21:51:59.716540 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.716482    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f2r7n" podStartSLOduration=34.688602887 podStartE2EDuration="46.716467136s" totalImagesPullingTime="12.027864249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:58.716063486 +0000 UTC m=+301.497961335" watchObservedRunningTime="2026-07-16 21:51:59.716467136 +0000 UTC m=+302.498364984"
Jul 16 21:51:59.766655 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:51:59.766623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:00.136956 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:00.136855    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.057866 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.057797    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9pdpx" podStartSLOduration=4.378818765 podStartE2EDuration="7.057762027s" totalImagesPullingTime="2.606448449s" totalInitContainerRuntime="72.494813ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:59.716601437 +0000 UTC m=+302.498499285" watchObservedRunningTime="2026-07-16 21:52:01.057762027 +0000 UTC m=+303.839659877"
Jul 16 21:52:01.058845 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.058813    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:52:01.062992 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.062964    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.067285 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.067259    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 21:52:01.067410 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.067393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-76rks\""
Jul 16 21:52:01.067656 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.067634    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 21:52:01.067787 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.067672    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 21:52:01.068125 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.068112    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 21:52:01.069255 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.069236    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 21:52:01.069365 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.069299    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 21:52:01.069801 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.069778    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 21:52:01.069801 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.069780    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 21:52:01.070012 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.069838    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 21:52:01.070075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.070053    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 21:52:01.070714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.070158    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 21:52:01.070714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.070298    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f80m22vpfbdbc\""
Jul 16 21:52:01.070714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.070407    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 21:52:01.070714 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.070584    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 21:52:01.071532 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.071511    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 21:52:01.074441 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.074418    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 21:52:01.088125 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.088092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:52:01.096527 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.096503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.179129 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179129 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-config\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc710204-7ef3-4629-ac3e-714d5962cc56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc710204-7ef3-4629-ac3e-714d5962cc56-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.179704 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdlx\" (UniqueName: \"kubernetes.io/projected/fc710204-7ef3-4629-ac3e-714d5962cc56-kube-api-access-sxdlx\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.180163 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.180163 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.179699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.280872 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.280840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.280872 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.280880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.280903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.280974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281130 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-config\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc710204-7ef3-4629-ac3e-714d5962cc56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc710204-7ef3-4629-ac3e-714d5962cc56-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.281474 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdlx\" (UniqueName: \"kubernetes.io/projected/fc710204-7ef3-4629-ac3e-714d5962cc56-kube-api-access-sxdlx\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.282057 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.281569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.283685 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.282325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.283685 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.282952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.283685 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.283354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.284696 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.284582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.285058 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.284855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.285148 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.285102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.285496 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.285470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.285610 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.285505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.285999 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.285972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc710204-7ef3-4629-ac3e-714d5962cc56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.287319 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.287292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.289654 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.289391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc710204-7ef3-4629-ac3e-714d5962cc56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.289654 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.289538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.289654 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.289617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc710204-7ef3-4629-ac3e-714d5962cc56-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.289886 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.289863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.289951 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.289927    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-config\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.290012 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.289991    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.290012 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.290001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.290407 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.290387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc710204-7ef3-4629-ac3e-714d5962cc56-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.292826 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.292804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdlx\" (UniqueName: \"kubernetes.io/projected/fc710204-7ef3-4629-ac3e-714d5962cc56-kube-api-access-sxdlx\") pod \"prometheus-k8s-0\" (UID: \"fc710204-7ef3-4629-ac3e-714d5962cc56\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.350179 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.350088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.380154 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.380118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:01.463774 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.463707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.530458 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.530418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:52:01.535936 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:52:01.535905    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc710204_7ef3_4629_ac3e_714d5962cc56.slice/crio-c58b90ebc26bed6f29d6f2623de09835e14661f9c886010248990e78ed32c499 WatchSource:0}: Error finding container c58b90ebc26bed6f29d6f2623de09835e14661f9c886010248990e78ed32c499: Status 404 returned error can't find the container with id c58b90ebc26bed6f29d6f2623de09835e14661f9c886010248990e78ed32c499
Jul 16 21:52:01.539589 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.539462    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:52:01.687202 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.687173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"c58b90ebc26bed6f29d6f2623de09835e14661f9c886010248990e78ed32c499"}
Jul 16 21:52:01.795763 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:01.795718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:02.692577 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:02.692523    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc710204-7ef3-4629-ac3e-714d5962cc56" containerID="a944b877ca94e9c061f501a1c48a9c25181ffdb7eb826634316493c589400663" exitCode=0
Jul 16 21:52:02.692922 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:02.692609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerDied","Data":"a944b877ca94e9c061f501a1c48a9c25181ffdb7eb826634316493c589400663"}
Jul 16 21:52:03.897361 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:03.897289    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-855975bf95-dpx9k" podUID="5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" containerName="registry" containerID="cri-o://2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0" gracePeriod=30
Jul 16 21:52:04.154418 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.154392    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:52:04.311924 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.311892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-installation-pull-secrets\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.311937    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-image-registry-private-configuration\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.311976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-bound-sa-token\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312008    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-ca-trust-extracted\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312067    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312300 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312095    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-trusted-ca\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312300 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312144    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf9rq\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-kube-api-access-nf9rq\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312300 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312170    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-certificates\") pod \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\" (UID: \"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3\") "
Jul 16 21:52:04.312830 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:04.312970 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.312923    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:04.314821 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.314791    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:04.315115 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.315063    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:04.315287 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.315259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:04.315396 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.315370    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:04.315568 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.315526    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-kube-api-access-nf9rq" (OuterVolumeSpecName: "kube-api-access-nf9rq") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "kube-api-access-nf9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:04.323905 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.323876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" (UID: "5a77c88f-42c2-4639-b425-7a7c1a5fe1e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:52:04.413038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.412952    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-trusted-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.412986    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9rq\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-kube-api-access-nf9rq\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.413000    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-certificates\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.413012    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-installation-pull-secrets\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.413026    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-image-registry-private-configuration\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413038 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.413039    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-bound-sa-token\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413336 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.413050    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-ca-trust-extracted\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.413336 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.413061    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3-registry-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:52:04.701978 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.701891    2562 generic.go:356] "Generic (PLEG): container finished" podID="5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" containerID="2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0" exitCode=0
Jul 16 21:52:04.701978 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.701936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-855975bf95-dpx9k" event={"ID":"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3","Type":"ContainerDied","Data":"2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0"}
Jul 16 21:52:04.701978 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.701955    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-855975bf95-dpx9k"
Jul 16 21:52:04.702226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.701984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-855975bf95-dpx9k" event={"ID":"5a77c88f-42c2-4639-b425-7a7c1a5fe1e3","Type":"ContainerDied","Data":"0c9589f350315ce23262fe960258a91fd9fae76356a8d3acba6f86bd5d393190"}
Jul 16 21:52:04.702226 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.702011    2562 scope.go:117] "RemoveContainer" containerID="2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0"
Jul 16 21:52:04.724373 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.724349    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-855975bf95-dpx9k"]
Jul 16 21:52:04.728819 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:04.728791    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-855975bf95-dpx9k"]
Jul 16 21:52:05.638205 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:05.638062    2562 scope.go:117] "RemoveContainer" containerID="2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0"
Jul 16 21:52:05.638465 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:52:05.638425    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0\": container with ID starting with 2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0 not found: ID does not exist" containerID="2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0"
Jul 16 21:52:05.638529 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:05.638462    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0"} err="failed to get container status \"2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0\": rpc error: code = NotFound desc = could not find container \"2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0\": container with ID starting with 2831fc7841cec0836370e3a0edd012eea4fbd3562f047fa126c79b162ccfece0 not found: ID does not exist"
Jul 16 21:52:05.745770 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:05.745744    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:52:05.750067 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:05.750012    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" path="/var/lib/kubelet/pods/5a77c88f-42c2-4639-b425-7a7c1a5fe1e3/volumes"
Jul 16 21:52:06.714192 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:06.714151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"911e638577fdbd331c5bf4beff83622bd2eecf9c72153d4cf755bd183a4be955"}
Jul 16 21:52:06.714593 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:06.714278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"2afd29feeff6339f96243158520a63f9d6f508133684dd62e1571a921c8cb0ea"}
Jul 16 21:52:07.720643 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:07.720615    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"cdf9057fbc2d0f2881d44421824bbed07a2db5f06b0094ad4856f572eb1d2ce9"}
Jul 16 21:52:07.720951 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:07.720679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"a8025166b58abb89fdb7112daadd12653eada6300316d3d3626fdb3706a96ce0"}
Jul 16 21:52:08.058922 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:08.058881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:08.121116 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:08.121082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:08.727920 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:08.727890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"4caf60649635ca36a1f027d9943fe33fd798d83cfb2311d684f8a246c3a78a5b"}
Jul 16 21:52:08.728294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:08.727952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc710204-7ef3-4629-ac3e-714d5962cc56","Type":"ContainerStarted","Data":"3c782b3ccdc51eba32a27779ada24ea080e715b46ccfac0f371066f86fbe9604"}
Jul 16 21:52:08.759707 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:08.759661    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.057137174 podStartE2EDuration="7.75964024s" totalImagesPullingTime="5.684554047s" totalInitContainerRuntime="17.949019ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:08.757994761 +0000 UTC m=+311.539892608" watchObservedRunningTime="2026-07-16 21:52:08.75964024 +0000 UTC m=+311.541538093"
Jul 16 21:52:10.274779 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:10.274742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:10.402117 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:10.402078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:10.751051 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:10.751028    2562 scope.go:117] "RemoveContainer" containerID="f17127a7f08b3cd9a1a4dd0d24469e0777aa2572df3d37e55585eb026d69b45b"
Jul 16 21:52:11.272448 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:11.272378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:11.380833 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:11.380801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:11.402851 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:11.402820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:11.741113 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:11.741082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d6c65b658-dl78l" event={"ID":"9c055adf-2e1e-4caf-86f1-c73b58e7a860","Type":"ContainerStarted","Data":"1094fafaf93228e50744212eb7dcb47b5b5c30a4211472f071e5605206561f21"}
Jul 16 21:52:12.285966 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:12.285936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:12.389309 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:12.389273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:13.128027 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:13.127994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:13.244572 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:13.244509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:13.416573 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:13.416519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:13.537769 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:13.537718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:13.634925 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:13.634890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:14.102758 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:14.102730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:14.207937 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:14.207904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:17.762937 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:17.762905    2562 generic.go:356] "Generic (PLEG): container finished" podID="76e3df6b-688c-4e41-ae3a-147f7febee41" containerID="db843f6e41bdeb4c00da46545f22ffec287a51c17b34682ae86839b85338f7f0" exitCode=0
Jul 16 21:52:17.763349 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:17.762950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj" event={"ID":"76e3df6b-688c-4e41-ae3a-147f7febee41","Type":"ContainerDied","Data":"db843f6e41bdeb4c00da46545f22ffec287a51c17b34682ae86839b85338f7f0"}
Jul 16 21:52:17.763349 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:17.763240    2562 scope.go:117] "RemoveContainer" containerID="db843f6e41bdeb4c00da46545f22ffec287a51c17b34682ae86839b85338f7f0"
Jul 16 21:52:18.768542 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:18.768512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7t9gj" event={"ID":"76e3df6b-688c-4e41-ae3a-147f7febee41","Type":"ContainerStarted","Data":"9803839113fcc3609ec3332778d5acc738051f741397e673071669a4ffb3d60a"}
Jul 16 21:52:28.408921 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:28.408886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:28.542958 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:28.542902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:29.287055 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:29.287031    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ds5ls_4dbdd52d-ece0-471e-8645-b79df023e5ac/serve-healthcheck-canary/0.log"
Jul 16 21:52:30.461315 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:30.461287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:30.520096 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:30.520060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:30.864821 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:30.864707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:33.604883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:33.604844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:33.739748 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:33.739714    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:33.829799 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:52:33.829767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:01.380885 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:01.380845    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:53:01.401001 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:01.400975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:53:01.722289 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:01.722233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:01.792393 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:01.792359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:01.940472 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:01.940445    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:53:09.542744 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:09.542712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:09.623409 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:09.623374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:11.667750 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:11.667713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:53:11.670172 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:11.670151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a2d502-339d-4f01-83ee-966860bc7a48-metrics-certs\") pod \"network-metrics-daemon-kbs6s\" (UID: \"10a2d502-339d-4f01-83ee-966860bc7a48\") " pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:53:11.749085 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:11.749055    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gkkkc\""
Jul 16 21:53:11.757441 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:11.757416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbs6s"
Jul 16 21:53:11.911491 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:11.911459    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kbs6s"]
Jul 16 21:53:11.914600 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:53:11.914567    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a2d502_339d_4f01_83ee_966860bc7a48.slice/crio-e20b8707f7c83fb4f12f25fd204f9204f68316e9886f131587f77fb1cc61d88b WatchSource:0}: Error finding container e20b8707f7c83fb4f12f25fd204f9204f68316e9886f131587f77fb1cc61d88b: Status 404 returned error can't find the container with id e20b8707f7c83fb4f12f25fd204f9204f68316e9886f131587f77fb1cc61d88b
Jul 16 21:53:11.962538 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:11.962505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbs6s" event={"ID":"10a2d502-339d-4f01-83ee-966860bc7a48","Type":"ContainerStarted","Data":"e20b8707f7c83fb4f12f25fd204f9204f68316e9886f131587f77fb1cc61d88b"}
Jul 16 21:53:12.967190 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:12.967167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbs6s" event={"ID":"10a2d502-339d-4f01-83ee-966860bc7a48","Type":"ContainerStarted","Data":"56714e23bb2e64ec8a02f5da12ccccc0ebdda2edefcaa30324d3142c493fbdaf"}
Jul 16 21:53:13.972275 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:13.972231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbs6s" event={"ID":"10a2d502-339d-4f01-83ee-966860bc7a48","Type":"ContainerStarted","Data":"418bd13c10473db9fc9859ce8806195c472a8041d8f7ae91fee51074d25ffa1e"}
Jul 16 21:53:13.989058 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:13.989011    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kbs6s" podStartSLOduration=375.089736261 podStartE2EDuration="6m15.988995081s" totalImagesPullingTime="899.25882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:13.988688053 +0000 UTC m=+376.770585901" watchObservedRunningTime="2026-07-16 21:53:13.988995081 +0000 UTC m=+376.770892929"
Jul 16 21:53:14.137686 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:14.137649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:14.227629 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:14.227506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:14.378497 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:14.378465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:22.536900 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:22.536842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:22.677850 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:22.677810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:22.739703 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:22.739672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:23.277475 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:23.277438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:23.376858 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:23.376817    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:29.352435 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:29.352405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:29.417364 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:29.417331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:31.158210 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:31.158172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:31.223924 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:31.223892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:31.749183 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:31.749127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:31.845374 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:31.845343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:41.579075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:41.579039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:41.662045 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:41.662010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:41.762751 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:41.762672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:41.864968 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:41.864871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:58.424658 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:58.424623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:58.526509 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:58.526461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:59.404310 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:59.404266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:59.480697 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:53:59.480666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:17.095351 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:17.095323    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:17.349781 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:17.349661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:23.712417 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:23.712388    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:23.775115 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:23.775067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:39.029359 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:39.029328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:39.832058 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:39.832023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:42.906644 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:42.906611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:43.021626 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:43.021596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:44.690111 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:44.690074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:44.841692 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:44.841662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:46.871190 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:46.871153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:47.319055 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:47.319027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:49.630609 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:49.630576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:49.730944 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:49.730912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:55.081742 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:55.081693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:57.354294 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:57.354259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:57.452098 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:54:57.452055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:02.916929 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:02.916857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:03.019204 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:03.019132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:04.790401 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:04.790350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:09.186237 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:09.186193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:09.284406 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:09.284355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:20.302633 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:20.302588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:20.632930 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:20.632841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:20.835983 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:20.835952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:21.462721 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.462690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:21.573293 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.573260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:21.707354 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.707315    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"]
Jul 16 21:55:21.707677 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.707661    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" containerName="registry"
Jul 16 21:55:21.707771 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.707682    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" containerName="registry"
Jul 16 21:55:21.707771 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.707747    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a77c88f-42c2-4639-b425-7a7c1a5fe1e3" containerName="registry"
Jul 16 21:55:21.710531 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.710511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.715216 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.715161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 21:55:21.721759 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.721742    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q5qbl\""
Jul 16 21:55:21.730251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.730237    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 21:55:21.737894 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.737881    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 21:55:21.739749 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.739737    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 21:55:21.756059 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.756037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"]
Jul 16 21:55:21.760883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.760862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9debcaac-f977-4b23-840d-81dd2ab761d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.760982 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.760903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdkn\" (UniqueName: \"kubernetes.io/projected/9debcaac-f977-4b23-840d-81dd2ab761d8-kube-api-access-tcdkn\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.760982 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.760928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9debcaac-f977-4b23-840d-81dd2ab761d8-tmp\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.760982 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.760946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9debcaac-f977-4b23-840d-81dd2ab761d8-manifests\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.761122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.760995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9debcaac-f977-4b23-840d-81dd2ab761d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.861483 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.861450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9debcaac-f977-4b23-840d-81dd2ab761d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.861683 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.861503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdkn\" (UniqueName: \"kubernetes.io/projected/9debcaac-f977-4b23-840d-81dd2ab761d8-kube-api-access-tcdkn\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.861683 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.861529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9debcaac-f977-4b23-840d-81dd2ab761d8-tmp\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.861683 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.861564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9debcaac-f977-4b23-840d-81dd2ab761d8-manifests\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.861683 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.861582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9debcaac-f977-4b23-840d-81dd2ab761d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.862028 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.862008    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9debcaac-f977-4b23-840d-81dd2ab761d8-manifests\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.864049 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.864024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9debcaac-f977-4b23-840d-81dd2ab761d8-tmp\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.864159 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.864091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9debcaac-f977-4b23-840d-81dd2ab761d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.864159 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.864139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9debcaac-f977-4b23-840d-81dd2ab761d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.876615 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.876593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdkn\" (UniqueName: \"kubernetes.io/projected/9debcaac-f977-4b23-840d-81dd2ab761d8-kube-api-access-tcdkn\") pod \"opendatahub-operator-controller-manager-df6666d-vt7sp\" (UID: \"9debcaac-f977-4b23-840d-81dd2ab761d8\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:21.975992 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:21.975893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:22.020613 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.020583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:22.185748 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.185698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"]
Jul 16 21:55:22.187878 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.187841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:22.188910 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:55:22.188888    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9debcaac_f977_4b23_840d_81dd2ab761d8.slice/crio-a1fb1c780ee9d70c71d8a88cf1f9a3163a1355b84ab0d1deebe63bbcb7a373d3 WatchSource:0}: Error finding container a1fb1c780ee9d70c71d8a88cf1f9a3163a1355b84ab0d1deebe63bbcb7a373d3: Status 404 returned error can't find the container with id a1fb1c780ee9d70c71d8a88cf1f9a3163a1355b84ab0d1deebe63bbcb7a373d3
Jul 16 21:55:22.424744 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.424710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp" event={"ID":"9debcaac-f977-4b23-840d-81dd2ab761d8","Type":"ContainerStarted","Data":"a1fb1c780ee9d70c71d8a88cf1f9a3163a1355b84ab0d1deebe63bbcb7a373d3"}
Jul 16 21:55:22.437564 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.437520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:22.662404 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.662321    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:22.832081 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:22.831996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:25.438163 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:25.438127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp" event={"ID":"9debcaac-f977-4b23-840d-81dd2ab761d8","Type":"ContainerStarted","Data":"67b53dca93376e8f2fd3f45ff7eed21c1e7fbd2c21ffa673501d29b61e4657a3"}
Jul 16 21:55:26.443452 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:26.443361    2562 generic.go:356] "Generic (PLEG): container finished" podID="9debcaac-f977-4b23-840d-81dd2ab761d8" containerID="67b53dca93376e8f2fd3f45ff7eed21c1e7fbd2c21ffa673501d29b61e4657a3" exitCode=0
Jul 16 21:55:26.443452 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:26.443421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp" event={"ID":"9debcaac-f977-4b23-840d-81dd2ab761d8","Type":"ContainerDied","Data":"67b53dca93376e8f2fd3f45ff7eed21c1e7fbd2c21ffa673501d29b61e4657a3"}
Jul 16 21:55:27.449606 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:27.449542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp" event={"ID":"9debcaac-f977-4b23-840d-81dd2ab761d8","Type":"ContainerStarted","Data":"ce02f3efca9b1601176da39307ead386b59eba1e8f7c0a38aaa17f7f166ac240"}
Jul 16 21:55:27.450013 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:27.449711    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:27.556905 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:27.556839    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp" podStartSLOduration=2.351953295 podStartE2EDuration="6.556814004s" totalImagesPullingTime="3.171627982s" totalInitContainerRuntime="1.033232727s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:27.553435784 +0000 UTC m=+510.335333633" watchObservedRunningTime="2026-07-16 21:55:27.556814004 +0000 UTC m=+510.338711852"
Jul 16 21:55:29.590884 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:29.590850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:29.864883 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:29.864798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:38.455405 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:38.455377    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-vt7sp"
Jul 16 21:55:41.770600 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:41.770530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:42.044787 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:42.044709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:42.750356 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:42.750322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:43.061315 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.061231    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"]
Jul 16 21:55:43.072022 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.071998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.118235 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.118205    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 21:55:43.118531 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.118518    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 21:55:43.119026 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.119008    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 21:55:43.125261 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.125238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"]
Jul 16 21:55:43.125876 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.125862    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pgr5g\""
Jul 16 21:55:43.132429 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.132409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98626467-5488-41b5-a3f5-498667623a37-tmp\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.132498 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.132453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98626467-5488-41b5-a3f5-498667623a37-tls-certs\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.132538 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.132492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzj9\" (UniqueName: \"kubernetes.io/projected/98626467-5488-41b5-a3f5-498667623a37-kube-api-access-wmzj9\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.143674 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.143654    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 21:55:43.233400 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.233359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98626467-5488-41b5-a3f5-498667623a37-tls-certs\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.233589 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.233412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzj9\" (UniqueName: \"kubernetes.io/projected/98626467-5488-41b5-a3f5-498667623a37-kube-api-access-wmzj9\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.233589 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.233463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98626467-5488-41b5-a3f5-498667623a37-tmp\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.235739 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.235711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98626467-5488-41b5-a3f5-498667623a37-tmp\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.235937 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.235918    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98626467-5488-41b5-a3f5-498667623a37-tls-certs\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.364501 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.364421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzj9\" (UniqueName: \"kubernetes.io/projected/98626467-5488-41b5-a3f5-498667623a37-kube-api-access-wmzj9\") pod \"kube-auth-proxy-5c8bcbdbf8-76tr8\" (UID: \"98626467-5488-41b5-a3f5-498667623a37\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.381278 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.381245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"
Jul 16 21:55:43.429316 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.429280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:43.672133 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:55:43.672102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98626467_5488_41b5_a3f5_498667623a37.slice/crio-32bd512bf6f86fba1463e42fd4a229189aaed4a430f15f055527f8073d17ebac WatchSource:0}: Error finding container 32bd512bf6f86fba1463e42fd4a229189aaed4a430f15f055527f8073d17ebac: Status 404 returned error can't find the container with id 32bd512bf6f86fba1463e42fd4a229189aaed4a430f15f055527f8073d17ebac
Jul 16 21:55:43.712695 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.712666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8"]
Jul 16 21:55:43.919309 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:43.919232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:44.155064 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:44.155030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:44.513627 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:44.513588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8" event={"ID":"98626467-5488-41b5-a3f5-498667623a37","Type":"ContainerStarted","Data":"32bd512bf6f86fba1463e42fd4a229189aaed4a430f15f055527f8073d17ebac"}
Jul 16 21:55:44.538761 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:44.538732    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:44.872383 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:44.872270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:46.120156 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:46.120130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:46.523129 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:46.523096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8" event={"ID":"98626467-5488-41b5-a3f5-498667623a37","Type":"ContainerStarted","Data":"2812781aa2d56d20cb66bd24d4c4abd70aebad23b90543cd55de4275eb80a076"}
Jul 16 21:55:46.604718 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:46.604667    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-76tr8" podStartSLOduration=2.182712268 podStartE2EDuration="4.60465189s" totalImagesPullingTime="2.421939622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:46.603102478 +0000 UTC m=+529.385000330" watchObservedRunningTime="2026-07-16 21:55:46.60465189 +0000 UTC m=+529.386549786"
Jul 16 21:55:46.756696 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:46.756666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:49.504366 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:49.504285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:49.841984 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:49.841894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:54.268961 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.268932    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"]
Jul 16 21:55:54.272459 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.272439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.275290 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.275271    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 21:55:54.275777 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.275764    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 21:55:54.276350 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.276332    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 21:55:54.276495 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.276373    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:55:54.276495 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.276413    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kt5xj\""
Jul 16 21:55:54.276905 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.276890    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 21:55:54.377872 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.377841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"]
Jul 16 21:55:54.440890 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.440855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxhh\" (UniqueName: \"kubernetes.io/projected/554f6cf4-30e2-434b-98bb-bc9ef8393691-kube-api-access-6nxhh\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.441087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.440900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/554f6cf4-30e2-434b-98bb-bc9ef8393691-manager-config\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.441087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.440928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f6cf4-30e2-434b-98bb-bc9ef8393691-metrics-cert\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.441087 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.441000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/554f6cf4-30e2-434b-98bb-bc9ef8393691-cert\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.541917 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.541833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxhh\" (UniqueName: \"kubernetes.io/projected/554f6cf4-30e2-434b-98bb-bc9ef8393691-kube-api-access-6nxhh\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.541917 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.541877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/554f6cf4-30e2-434b-98bb-bc9ef8393691-manager-config\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.541917 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.541903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f6cf4-30e2-434b-98bb-bc9ef8393691-metrics-cert\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.542145 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.542025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/554f6cf4-30e2-434b-98bb-bc9ef8393691-cert\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.542627 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.542603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/554f6cf4-30e2-434b-98bb-bc9ef8393691-manager-config\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.544566 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.544526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/554f6cf4-30e2-434b-98bb-bc9ef8393691-cert\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.544566 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.544532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/554f6cf4-30e2-434b-98bb-bc9ef8393691-metrics-cert\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.566021 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.565996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxhh\" (UniqueName: \"kubernetes.io/projected/554f6cf4-30e2-434b-98bb-bc9ef8393691-kube-api-access-6nxhh\") pod \"lws-controller-manager-6c6c96995f-xmgpv\" (UID: \"554f6cf4-30e2-434b-98bb-bc9ef8393691\") " pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.581865 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.581838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:55:54.737914 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:54.737886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"]
Jul 16 21:55:54.741017 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:55:54.740993    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554f6cf4_30e2_434b_98bb_bc9ef8393691.slice/crio-c539b849f1d25d6e9cf07e95f4c515591911e310e786bbd078935f853e25ed63 WatchSource:0}: Error finding container c539b849f1d25d6e9cf07e95f4c515591911e310e786bbd078935f853e25ed63: Status 404 returned error can't find the container with id c539b849f1d25d6e9cf07e95f4c515591911e310e786bbd078935f853e25ed63
Jul 16 21:55:55.240267 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:55.240231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.460634 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:55.460599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.558957 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:55.558878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv" event={"ID":"554f6cf4-30e2-434b-98bb-bc9ef8393691","Type":"ContainerStarted","Data":"c539b849f1d25d6e9cf07e95f4c515591911e310e786bbd078935f853e25ed63"}
Jul 16 21:55:55.735574 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:55.735512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:57.044870 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:57.044838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:57.576734 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:57.576702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:57.948757 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:57.948722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:59.881054 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:55:59.881020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:00.207939 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:00.207910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:00.757188 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:00.757152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:03.886252 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:03.886200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:04.172693 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:04.172661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:04.418735 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:04.418703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:07.610946 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:07.610910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv" event={"ID":"554f6cf4-30e2-434b-98bb-bc9ef8393691","Type":"ContainerStarted","Data":"aa08a31cc0f8da4935455c18c11822cb0797effd72d343f842cf89e6b886e095"}
Jul 16 21:56:07.611316 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:07.611057    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:56:18.617472 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:18.617436    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv"
Jul 16 21:56:18.644111 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:18.644053    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c6c96995f-xmgpv" podStartSLOduration=12.310292243 podStartE2EDuration="24.644033091s" totalImagesPullingTime="12.333740848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:07.703743956 +0000 UTC m=+550.485641805" watchObservedRunningTime="2026-07-16 21:56:18.644033091 +0000 UTC m=+561.425930942"
Jul 16 21:56:23.848486 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:23.848450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:34.903065 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:34.903030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:43.056463 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:43.056432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:56:44.633323 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:56:44.633287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:14.361251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:14.361214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:14.476736 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:14.476704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:16.496894 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:16.496857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:16.580629 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:16.580596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:18.620062 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:18.619978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:24.219180 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:24.219148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:24.816918 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:24.816882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:25.656113 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:25.656081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:25.757197 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:25.757165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:28.313619 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:28.313584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:28.618760 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:28.618675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:30.094127 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.094096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:30.630342 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.630314    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"]
Jul 16 21:57:30.633467 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.633446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:30.636071 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.636053    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:57:30.636136 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.636101    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:57:30.636869 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.636854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qw429\""
Jul 16 21:57:30.645274 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.645249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"]
Jul 16 21:57:30.739463 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.739432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6t4\" (UniqueName: \"kubernetes.io/projected/cd8ce30b-e824-4175-9ef2-70158d1ac334-kube-api-access-ck6t4\") pod \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" (UID: \"cd8ce30b-e824-4175-9ef2-70158d1ac334\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:30.840873 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.840835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6t4\" (UniqueName: \"kubernetes.io/projected/cd8ce30b-e824-4175-9ef2-70158d1ac334-kube-api-access-ck6t4\") pod \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" (UID: \"cd8ce30b-e824-4175-9ef2-70158d1ac334\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:30.855867 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.855835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6t4\" (UniqueName: \"kubernetes.io/projected/cd8ce30b-e824-4175-9ef2-70158d1ac334-kube-api-access-ck6t4\") pod \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" (UID: \"cd8ce30b-e824-4175-9ef2-70158d1ac334\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:30.943629 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:30.943599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:31.076607 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.076579    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"]
Jul 16 21:57:31.080251 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:57:31.080223    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8ce30b_e824_4175_9ef2_70158d1ac334.slice/crio-52b1bf754a532c0918781793c6b893c5152eb1d63cbe39f3a8bbae09e21a8a7a WatchSource:0}: Error finding container 52b1bf754a532c0918781793c6b893c5152eb1d63cbe39f3a8bbae09e21a8a7a: Status 404 returned error can't find the container with id 52b1bf754a532c0918781793c6b893c5152eb1d63cbe39f3a8bbae09e21a8a7a
Jul 16 21:57:31.082347 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.082328    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:57:31.361383 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.361307    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"]
Jul 16 21:57:31.361930 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.361908    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:31.365596 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.365582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.369450 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.369422    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pm9j9\""
Jul 16 21:57:31.369450 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.369426    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 21:57:31.369655 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.369560    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 21:57:31.378903 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.378871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"]
Jul 16 21:57:31.446227 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.446197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a00aeffd-1a16-4497-a925-b660f84b0976-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.446387 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.446235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a00aeffd-1a16-4497-a925-b660f84b0976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.446387 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.446300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9z4b\" (UniqueName: \"kubernetes.io/projected/a00aeffd-1a16-4497-a925-b660f84b0976-kube-api-access-m9z4b\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.547447 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.547408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a00aeffd-1a16-4497-a925-b660f84b0976-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.547622 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.547464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a00aeffd-1a16-4497-a925-b660f84b0976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.547622 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.547508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9z4b\" (UniqueName: \"kubernetes.io/projected/a00aeffd-1a16-4497-a925-b660f84b0976-kube-api-access-m9z4b\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.547718 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:31.547639    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 16 21:57:31.547813 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:31.547731    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00aeffd-1a16-4497-a925-b660f84b0976-plugin-serving-cert podName:a00aeffd-1a16-4497-a925-b660f84b0976 nodeName:}" failed. No retries permitted until 2026-07-16 21:57:32.047707153 +0000 UTC m=+634.829604980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a00aeffd-1a16-4497-a925-b660f84b0976-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-5q7z6" (UID: "a00aeffd-1a16-4497-a925-b660f84b0976") : secret "plugin-serving-cert" not found
Jul 16 21:57:31.548153 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.548134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a00aeffd-1a16-4497-a925-b660f84b0976-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.563918 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.563893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9z4b\" (UniqueName: \"kubernetes.io/projected/a00aeffd-1a16-4497-a925-b660f84b0976-kube-api-access-m9z4b\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:31.634854 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.634771    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:31.744879 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.744842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:31.920470 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:31.920431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt" event={"ID":"cd8ce30b-e824-4175-9ef2-70158d1ac334","Type":"ContainerStarted","Data":"52b1bf754a532c0918781793c6b893c5152eb1d63cbe39f3a8bbae09e21a8a7a"}
Jul 16 21:57:32.052475 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.052424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a00aeffd-1a16-4497-a925-b660f84b0976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:32.055202 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.055168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a00aeffd-1a16-4497-a925-b660f84b0976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5q7z6\" (UID: \"a00aeffd-1a16-4497-a925-b660f84b0976\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:32.275127 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.275031    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"
Jul 16 21:57:32.405443 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.405367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6"]
Jul 16 21:57:32.406918 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:57:32.406888    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda00aeffd_1a16_4497_a925_b660f84b0976.slice/crio-8940454222b9a616682eaea1c5defedac99a7727e131c2cfd67c1552566d4be4 WatchSource:0}: Error finding container 8940454222b9a616682eaea1c5defedac99a7727e131c2cfd67c1552566d4be4: Status 404 returned error can't find the container with id 8940454222b9a616682eaea1c5defedac99a7727e131c2cfd67c1552566d4be4
Jul 16 21:57:32.863820 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.863792    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:32.927242 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.927202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt" event={"ID":"cd8ce30b-e824-4175-9ef2-70158d1ac334","Type":"ContainerStarted","Data":"0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69"}
Jul 16 21:57:32.927496 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.927332    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:32.928565 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.928517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6" event={"ID":"a00aeffd-1a16-4497-a925-b660f84b0976","Type":"ContainerStarted","Data":"8940454222b9a616682eaea1c5defedac99a7727e131c2cfd67c1552566d4be4"}
Jul 16 21:57:32.945940 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:32.945889    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt" podStartSLOduration=1.228855597 podStartE2EDuration="2.945872093s" totalImagesPullingTime="1.717016496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:32.944122876 +0000 UTC m=+635.726020726" watchObservedRunningTime="2026-07-16 21:57:32.945872093 +0000 UTC m=+635.727769941"
Jul 16 21:57:34.956043 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:34.956011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:35.013111 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:35.013079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:35.416894 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:35.416862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:35.518104 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:35.518072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:43.011105 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:43.011020    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"]
Jul 16 21:57:43.011572 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:43.011244    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" containerName="manager" containerID="cri-o://0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69" gracePeriod=2
Jul 16 21:57:43.013342 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:43.013296    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:43.021231 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:43.019449    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"]
Jul 16 21:57:43.032115 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:43.032071    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" is forbidden: User \"system:node:ip-10-0-139-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-156.ec2.internal' and this object" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:44.275416 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.275171    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"]
Jul 16 21:57:44.275898 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.275597    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" containerName="manager"
Jul 16 21:57:44.275898 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.275613    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" containerName="manager"
Jul 16 21:57:44.275898 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.275687    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" containerName="manager"
Jul 16 21:57:44.317972 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.317939    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"]
Jul 16 21:57:44.318161 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.318045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.320858 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.320831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ld9cr\""
Jul 16 21:57:44.464143 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.464103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wjw26\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.464143 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.464148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrmt\" (UniqueName: \"kubernetes.io/projected/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-kube-api-access-gdrmt\") pod \"kuadrant-operator-controller-manager-86878c769f-wjw26\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.565277 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.565183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wjw26\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.565277 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.565238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrmt\" (UniqueName: \"kubernetes.io/projected/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-kube-api-access-gdrmt\") pod \"kuadrant-operator-controller-manager-86878c769f-wjw26\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.565658 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.565624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wjw26\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.577620 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.577567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrmt\" (UniqueName: \"kubernetes.io/projected/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-kube-api-access-gdrmt\") pod \"kuadrant-operator-controller-manager-86878c769f-wjw26\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:44.628950 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:44.628918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:45.449985 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:45.449822    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:45.548357 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:45.548318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:48.458523 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:48.458485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:48.559447 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:48.559404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:50.889784 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:50.889750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:50.992016 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:50.991980    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:53.777650 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:53.777619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"]
Jul 16 21:57:53.781633 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:57:53.781609    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9279f9e8_d36b_45c3_a2b9_36ccc6a09df7.slice/crio-49184aee3a0bb5dea125ffa432f11af237e3fe46b0652692ef77f9718fc7eadb WatchSource:0}: Error finding container 49184aee3a0bb5dea125ffa432f11af237e3fe46b0652692ef77f9718fc7eadb: Status 404 returned error can't find the container with id 49184aee3a0bb5dea125ffa432f11af237e3fe46b0652692ef77f9718fc7eadb
Jul 16 21:57:53.824863 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:53.824836    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:53.827059 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:53.827030    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" is forbidden: User \"system:node:ip-10-0-139-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-156.ec2.internal' and this object" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:53.853175 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:53.853143    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck6t4\" (UniqueName: \"kubernetes.io/projected/cd8ce30b-e824-4175-9ef2-70158d1ac334-kube-api-access-ck6t4\") pod \"cd8ce30b-e824-4175-9ef2-70158d1ac334\" (UID: \"cd8ce30b-e824-4175-9ef2-70158d1ac334\") "
Jul 16 21:57:53.855418 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:53.855383    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8ce30b-e824-4175-9ef2-70158d1ac334-kube-api-access-ck6t4" (OuterVolumeSpecName: "kube-api-access-ck6t4") pod "cd8ce30b-e824-4175-9ef2-70158d1ac334" (UID: "cd8ce30b-e824-4175-9ef2-70158d1ac334"). InnerVolumeSpecName "kube-api-access-ck6t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:53.954035 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:53.954006    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6t4\" (UniqueName: \"kubernetes.io/projected/cd8ce30b-e824-4175-9ef2-70158d1ac334-kube-api-access-ck6t4\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:57:54.028037 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:54.028002    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd8ce30b-e824-4175-9ef2-70158d1ac334" containerID="0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69" exitCode=0
Jul 16 21:57:54.028202 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:54.028050    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:54.028202 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:54.028109    2562 scope.go:117] "RemoveContainer" containerID="0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69"
Jul 16 21:57:54.029660 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:54.029636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26" event={"ID":"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7","Type":"ContainerStarted","Data":"49184aee3a0bb5dea125ffa432f11af237e3fe46b0652692ef77f9718fc7eadb"}
Jul 16 21:57:54.030396 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:54.030377    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" is forbidden: User \"system:node:ip-10-0-139-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-156.ec2.internal' and this object" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:54.037737 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:54.037719    2562 scope.go:117] "RemoveContainer" containerID="0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69"
Jul 16 21:57:54.038005 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:54.037985    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69\": container with ID starting with 0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69 not found: ID does not exist" containerID="0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69"
Jul 16 21:57:54.038065 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:54.038011    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69"} err="failed to get container status \"0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69\": rpc error: code = NotFound desc = could not find container \"0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69\": container with ID starting with 0ed866c658c98c2d2f9dc273e8b6c86656883bda5184265bc83b49416a0a0f69 not found: ID does not exist"
Jul 16 21:57:54.038662 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:54.038642    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" is forbidden: User \"system:node:ip-10-0-139-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-156.ec2.internal' and this object" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:55.037131 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:55.037096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6" event={"ID":"a00aeffd-1a16-4497-a925-b660f84b0976","Type":"ContainerStarted","Data":"3b3448c19f3229c27c20dd70b6ff67ad47bb4a8f6146d38449f6d52a79323089"}
Jul 16 21:57:55.039288 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:57:55.039256    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nbpwt\" is forbidden: User \"system:node:ip-10-0-139-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-156.ec2.internal' and this object" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbpwt"
Jul 16 21:57:55.054314 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:55.054246    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5q7z6" podStartSLOduration=2.4101691499999998 podStartE2EDuration="24.054228125s" totalImagesPullingTime="21.644058975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:55.053990854 +0000 UTC m=+657.835888703" watchObservedRunningTime="2026-07-16 21:57:55.054228125 +0000 UTC m=+657.836125974"
Jul 16 21:57:55.752804 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:55.752762    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8ce30b-e824-4175-9ef2-70158d1ac334" path="/var/lib/kubelet/pods/cd8ce30b-e824-4175-9ef2-70158d1ac334/volumes"
Jul 16 21:57:56.997245 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:56.997201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:57.099115 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:57.099079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:59.062652 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:59.062612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26" event={"ID":"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7","Type":"ContainerStarted","Data":"151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9"}
Jul 16 21:57:59.063078 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:59.062724    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:57:59.082388 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:57:59.082343    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26" podStartSLOduration=10.728025869 podStartE2EDuration="15.082328067s" totalImagesPullingTime="4.354302198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:59.080198804 +0000 UTC m=+661.862096654" watchObservedRunningTime="2026-07-16 21:57:59.082328067 +0000 UTC m=+661.864225972"
Jul 16 21:58:05.851057 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:05.851023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:05.952231 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:05.952198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:08.847865 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:08.847822    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:08.944844 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:08.944787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:10.069622 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:10.069596    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 21:58:11.300923 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:11.300888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:11.401449 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:11.401409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:14.971250 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:14.971206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:20.815095 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:20.815048    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:22.193473 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:22.193435    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:26.555020 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:26.554981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:26.845947 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:26.845848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:27.260682 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:27.260646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:27.361862 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:27.361827    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:27.462645 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:27.462612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:27.938516 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:27.938478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:27.996122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:27.996077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:31.465164 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:31.465124    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:31.576006 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:31.575973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:31.636764 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:31.636735    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:32.290433 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.290395    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-v7w58"]
Jul 16 21:58:32.328325 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.328279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:32.335535 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.335513    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-v7w58"]
Jul 16 21:58:32.335675 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.335612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:32.338708 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.338687    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ftjlj\""
Jul 16 21:58:32.395885 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.395852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khl48\" (UniqueName: \"kubernetes.io/projected/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a-kube-api-access-khl48\") pod \"authorino-58cd955675-v7w58\" (UID: \"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a\") " pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:32.414195 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.414164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:32.497319 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.497280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khl48\" (UniqueName: \"kubernetes.io/projected/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a-kube-api-access-khl48\") pod \"authorino-58cd955675-v7w58\" (UID: \"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a\") " pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:32.507570 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.507019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khl48\" (UniqueName: \"kubernetes.io/projected/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a-kube-api-access-khl48\") pod \"authorino-58cd955675-v7w58\" (UID: \"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a\") " pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:32.645636 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.645538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:32.770355 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:32.770327    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-v7w58"]
Jul 16 21:58:32.772964 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:58:32.772937    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3d4d543_4eb7_4da5_9c57_6dafc3c0000a.slice/crio-2f18aec1e3513dcb5ed108f5dc1e9bafa5d21ebc32d1088bd39367a15a09df31 WatchSource:0}: Error finding container 2f18aec1e3513dcb5ed108f5dc1e9bafa5d21ebc32d1088bd39367a15a09df31: Status 404 returned error can't find the container with id 2f18aec1e3513dcb5ed108f5dc1e9bafa5d21ebc32d1088bd39367a15a09df31
Jul 16 21:58:33.210785 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:33.210750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v7w58" event={"ID":"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a","Type":"ContainerStarted","Data":"2f18aec1e3513dcb5ed108f5dc1e9bafa5d21ebc32d1088bd39367a15a09df31"}
Jul 16 21:58:33.655571 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:33.655508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:36.227167 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:36.227137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v7w58" event={"ID":"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a","Type":"ContainerStarted","Data":"39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199"}
Jul 16 21:58:36.241372 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:36.241329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-v7w58" podStartSLOduration=1.191630551 podStartE2EDuration="4.241316148s" totalImagesPullingTime="3.049685597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:36.240745274 +0000 UTC m=+699.022643122" watchObservedRunningTime="2026-07-16 21:58:36.241316148 +0000 UTC m=+699.023213996"
Jul 16 21:58:37.515832 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:37.515802    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-v7w58"]
Jul 16 21:58:38.235482 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.235438    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-v7w58" podUID="b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" containerName="authorino" containerID="cri-o://39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199" gracePeriod=30
Jul 16 21:58:38.479260 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.479235    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:38.550082 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.550003    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khl48\" (UniqueName: \"kubernetes.io/projected/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a-kube-api-access-khl48\") pod \"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a\" (UID: \"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a\") "
Jul 16 21:58:38.552141 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.552114    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a-kube-api-access-khl48" (OuterVolumeSpecName: "kube-api-access-khl48") pod "b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" (UID: "b3d4d543-4eb7-4da5-9c57-6dafc3c0000a"). InnerVolumeSpecName "kube-api-access-khl48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:58:38.651450 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.651412    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khl48\" (UniqueName: \"kubernetes.io/projected/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a-kube-api-access-khl48\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:58:38.718235 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.718202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:38.804030 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:38.803944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:39.240955 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.240926    2562 generic.go:356] "Generic (PLEG): container finished" podID="b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" containerID="39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199" exitCode=0
Jul 16 21:58:39.241122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.240968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v7w58" event={"ID":"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a","Type":"ContainerDied","Data":"39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199"}
Jul 16 21:58:39.241122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.240986    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v7w58"
Jul 16 21:58:39.241122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.240994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v7w58" event={"ID":"b3d4d543-4eb7-4da5-9c57-6dafc3c0000a","Type":"ContainerDied","Data":"2f18aec1e3513dcb5ed108f5dc1e9bafa5d21ebc32d1088bd39367a15a09df31"}
Jul 16 21:58:39.241122 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.241010    2562 scope.go:117] "RemoveContainer" containerID="39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199"
Jul 16 21:58:39.251718 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.251700    2562 scope.go:117] "RemoveContainer" containerID="39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199"
Jul 16 21:58:39.251990 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:58:39.251970    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199\": container with ID starting with 39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199 not found: ID does not exist" containerID="39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199"
Jul 16 21:58:39.252091 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.251996    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199"} err="failed to get container status \"39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199\": rpc error: code = NotFound desc = could not find container \"39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199\": container with ID starting with 39d6f4cc44dfd28ab882034cf39d781d8f68a9b0cd42af302e9d42bec42ec199 not found: ID does not exist"
Jul 16 21:58:39.261104 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.261082    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-v7w58"]
Jul 16 21:58:39.267463 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.267446    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-v7w58"]
Jul 16 21:58:39.750019 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:39.749991    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" path="/var/lib/kubelet/pods/b3d4d543-4eb7-4da5-9c57-6dafc3c0000a/volumes"
Jul 16 21:58:52.847430 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:52.847342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:52.945189 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:52.945162    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:53.368952 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:53.368912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:54.909075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:58:54.909044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:05.919898 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:05.919864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:05.977139 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:05.977104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:06.968269 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:06.968213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:07.171071 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.171034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:07.259262 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.259184    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4ckjc"]
Jul 16 21:59:07.259516 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.259503    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" containerName="authorino"
Jul 16 21:59:07.259592 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.259518    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" containerName="authorino"
Jul 16 21:59:07.259592 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.259587    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d4d543-4eb7-4da5-9c57-6dafc3c0000a" containerName="authorino"
Jul 16 21:59:07.262073 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.262055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:07.264457 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.264438    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ftjlj\""
Jul 16 21:59:07.268283 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.268262    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4ckjc"]
Jul 16 21:59:07.392578 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.392537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/bfb54cb0-5535-4338-aa2d-8a6013fae101-kube-api-access-r2kkv\") pod \"authorino-5d4b67c446-4ckjc\" (UID: \"bfb54cb0-5535-4338-aa2d-8a6013fae101\") " pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:07.493946 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.493904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/bfb54cb0-5535-4338-aa2d-8a6013fae101-kube-api-access-r2kkv\") pod \"authorino-5d4b67c446-4ckjc\" (UID: \"bfb54cb0-5535-4338-aa2d-8a6013fae101\") " pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:07.498733 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.498700    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4ckjc"]
Jul 16 21:59:07.498938 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:59:07.498920    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r2kkv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-4ckjc" podUID="bfb54cb0-5535-4338-aa2d-8a6013fae101"
Jul 16 21:59:07.508846 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.508818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/bfb54cb0-5535-4338-aa2d-8a6013fae101-kube-api-access-r2kkv\") pod \"authorino-5d4b67c446-4ckjc\" (UID: \"bfb54cb0-5535-4338-aa2d-8a6013fae101\") " pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:07.526191 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.526134    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-v69sc"]
Jul 16 21:59:07.528381 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.528363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:07.530968 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.530949    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 21:59:07.536754 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.536732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-v69sc"]
Jul 16 21:59:07.561816 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.561788    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-v69sc"]
Jul 16 21:59:07.562026 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:59:07.562007    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fqc4w tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-fqc4w tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-v69sc" podUID="d08a8026-e747-402c-a57d-4fec265fc8d5"
Jul 16 21:59:07.695118 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.695083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d08a8026-e747-402c-a57d-4fec265fc8d5-tls-cert\") pod \"authorino-76877994-v69sc\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") " pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:07.695277 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.695126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqc4w\" (UniqueName: \"kubernetes.io/projected/d08a8026-e747-402c-a57d-4fec265fc8d5-kube-api-access-fqc4w\") pod \"authorino-76877994-v69sc\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") " pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:07.795710 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.795623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d08a8026-e747-402c-a57d-4fec265fc8d5-tls-cert\") pod \"authorino-76877994-v69sc\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") " pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:07.795710 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.795664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc4w\" (UniqueName: \"kubernetes.io/projected/d08a8026-e747-402c-a57d-4fec265fc8d5-kube-api-access-fqc4w\") pod \"authorino-76877994-v69sc\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") " pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:07.798476 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.798453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d08a8026-e747-402c-a57d-4fec265fc8d5-tls-cert\") pod \"authorino-76877994-v69sc\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") " pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:07.803694 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:07.803668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc4w\" (UniqueName: \"kubernetes.io/projected/d08a8026-e747-402c-a57d-4fec265fc8d5-kube-api-access-fqc4w\") pod \"authorino-76877994-v69sc\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") " pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:08.358856 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.358822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:08.359297 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.358822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:08.365949 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.365926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:08.368663 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.368644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:08.502084 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.502055    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d08a8026-e747-402c-a57d-4fec265fc8d5-tls-cert\") pod \"d08a8026-e747-402c-a57d-4fec265fc8d5\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") "
Jul 16 21:59:08.502246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.502112    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqc4w\" (UniqueName: \"kubernetes.io/projected/d08a8026-e747-402c-a57d-4fec265fc8d5-kube-api-access-fqc4w\") pod \"d08a8026-e747-402c-a57d-4fec265fc8d5\" (UID: \"d08a8026-e747-402c-a57d-4fec265fc8d5\") "
Jul 16 21:59:08.502246 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.502152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/bfb54cb0-5535-4338-aa2d-8a6013fae101-kube-api-access-r2kkv\") pod \"bfb54cb0-5535-4338-aa2d-8a6013fae101\" (UID: \"bfb54cb0-5535-4338-aa2d-8a6013fae101\") "
Jul 16 21:59:08.504448 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.504414    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08a8026-e747-402c-a57d-4fec265fc8d5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d08a8026-e747-402c-a57d-4fec265fc8d5" (UID: "d08a8026-e747-402c-a57d-4fec265fc8d5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:59:08.504590 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.504457    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb54cb0-5535-4338-aa2d-8a6013fae101-kube-api-access-r2kkv" (OuterVolumeSpecName: "kube-api-access-r2kkv") pod "bfb54cb0-5535-4338-aa2d-8a6013fae101" (UID: "bfb54cb0-5535-4338-aa2d-8a6013fae101"). InnerVolumeSpecName "kube-api-access-r2kkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:59:08.504590 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.504477    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08a8026-e747-402c-a57d-4fec265fc8d5-kube-api-access-fqc4w" (OuterVolumeSpecName: "kube-api-access-fqc4w") pod "d08a8026-e747-402c-a57d-4fec265fc8d5" (UID: "d08a8026-e747-402c-a57d-4fec265fc8d5"). InnerVolumeSpecName "kube-api-access-fqc4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:59:08.602964 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.602931    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d08a8026-e747-402c-a57d-4fec265fc8d5-tls-cert\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:59:08.602964 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.602959    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqc4w\" (UniqueName: \"kubernetes.io/projected/d08a8026-e747-402c-a57d-4fec265fc8d5-kube-api-access-fqc4w\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:59:08.603145 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:08.602972    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2kkv\" (UniqueName: \"kubernetes.io/projected/bfb54cb0-5535-4338-aa2d-8a6013fae101-kube-api-access-r2kkv\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:59:09.362497 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.362468    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4ckjc"
Jul 16 21:59:09.362982 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.362469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-v69sc"
Jul 16 21:59:09.391893 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.391871    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-v69sc"]
Jul 16 21:59:09.395867 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.395837    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-v69sc"]
Jul 16 21:59:09.414508 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.414484    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4ckjc"]
Jul 16 21:59:09.418232 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.418207    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4ckjc"]
Jul 16 21:59:09.750584 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.750529    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb54cb0-5535-4338-aa2d-8a6013fae101" path="/var/lib/kubelet/pods/bfb54cb0-5535-4338-aa2d-8a6013fae101/volumes"
Jul 16 21:59:09.750814 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:09.750801    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08a8026-e747-402c-a57d-4fec265fc8d5" path="/var/lib/kubelet/pods/d08a8026-e747-402c-a57d-4fec265fc8d5/volumes"
Jul 16 21:59:14.858272 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.858236    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:14.882045 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.882016    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cdc9548dd-5gs7h"]
Jul 16 21:59:14.886232 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.886213    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:14.888707 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.888686    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 21:59:14.888707 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.888700    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sckmc\""
Jul 16 21:59:14.888841 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.888702    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 21:59:14.894359 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.894335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-5gs7h"]
Jul 16 21:59:14.974401 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:14.974368    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:15.031407 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.031375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:15.058953 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.058920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed2eded8-c029-4b25-9529-4432cb216130-webhook-cert\") pod \"maas-controller-cdc9548dd-5gs7h\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") " pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.059109 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.058956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8t2\" (UniqueName: \"kubernetes.io/projected/ed2eded8-c029-4b25-9529-4432cb216130-kube-api-access-6c8t2\") pod \"maas-controller-cdc9548dd-5gs7h\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") " pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.160320 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.160292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed2eded8-c029-4b25-9529-4432cb216130-webhook-cert\") pod \"maas-controller-cdc9548dd-5gs7h\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") " pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.160500 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.160325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8t2\" (UniqueName: \"kubernetes.io/projected/ed2eded8-c029-4b25-9529-4432cb216130-kube-api-access-6c8t2\") pod \"maas-controller-cdc9548dd-5gs7h\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") " pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.162889 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.162865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed2eded8-c029-4b25-9529-4432cb216130-webhook-cert\") pod \"maas-controller-cdc9548dd-5gs7h\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") " pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.168602 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.168577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8t2\" (UniqueName: \"kubernetes.io/projected/ed2eded8-c029-4b25-9529-4432cb216130-kube-api-access-6c8t2\") pod \"maas-controller-cdc9548dd-5gs7h\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") " pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.197348 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.197330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:15.324408 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.324375    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-5gs7h"]
Jul 16 21:59:15.327578 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:59:15.327525    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2eded8_c029_4b25_9529_4432cb216130.slice/crio-2a3fd120b0e37784e5ef7863a087b343f8f25055b046156a414ae686336fa730 WatchSource:0}: Error finding container 2a3fd120b0e37784e5ef7863a087b343f8f25055b046156a414ae686336fa730: Status 404 returned error can't find the container with id 2a3fd120b0e37784e5ef7863a087b343f8f25055b046156a414ae686336fa730
Jul 16 21:59:15.387897 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:15.387868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-5gs7h" event={"ID":"ed2eded8-c029-4b25-9529-4432cb216130","Type":"ContainerStarted","Data":"2a3fd120b0e37784e5ef7863a087b343f8f25055b046156a414ae686336fa730"}
Jul 16 21:59:18.097100 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:18.097062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:18.195803 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:18.195756    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:19.409146 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:19.409106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-5gs7h" event={"ID":"ed2eded8-c029-4b25-9529-4432cb216130","Type":"ContainerStarted","Data":"ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b"}
Jul 16 21:59:19.409635 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:19.409228    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:19.426888 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:19.426832    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cdc9548dd-5gs7h" podStartSLOduration=2.38386218 podStartE2EDuration="5.426819006s" totalImagesPullingTime="3.042956826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:19.42521122 +0000 UTC m=+742.207109070" watchObservedRunningTime="2026-07-16 21:59:19.426819006 +0000 UTC m=+742.208716854"
Jul 16 21:59:29.752686 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:29.752656    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-5gs7h"]
Jul 16 21:59:29.753146 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:29.752863    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cdc9548dd-5gs7h" podUID="ed2eded8-c029-4b25-9529-4432cb216130" containerName="manager" containerID="cri-o://ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b" gracePeriod=30
Jul 16 21:59:29.754785 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:29.754755    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:29.994143 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:29.994122    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:30.088779 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.088694    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c8t2\" (UniqueName: \"kubernetes.io/projected/ed2eded8-c029-4b25-9529-4432cb216130-kube-api-access-6c8t2\") pod \"ed2eded8-c029-4b25-9529-4432cb216130\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") "
Jul 16 21:59:30.088779 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.088733    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed2eded8-c029-4b25-9529-4432cb216130-webhook-cert\") pod \"ed2eded8-c029-4b25-9529-4432cb216130\" (UID: \"ed2eded8-c029-4b25-9529-4432cb216130\") "
Jul 16 21:59:30.091003 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.090956    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2eded8-c029-4b25-9529-4432cb216130-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ed2eded8-c029-4b25-9529-4432cb216130" (UID: "ed2eded8-c029-4b25-9529-4432cb216130"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:59:30.091003 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.090967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2eded8-c029-4b25-9529-4432cb216130-kube-api-access-6c8t2" (OuterVolumeSpecName: "kube-api-access-6c8t2") pod "ed2eded8-c029-4b25-9529-4432cb216130" (UID: "ed2eded8-c029-4b25-9529-4432cb216130"). InnerVolumeSpecName "kube-api-access-6c8t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:59:30.189719 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.189684    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8t2\" (UniqueName: \"kubernetes.io/projected/ed2eded8-c029-4b25-9529-4432cb216130-kube-api-access-6c8t2\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:59:30.189719 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.189712    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed2eded8-c029-4b25-9529-4432cb216130-webhook-cert\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 21:59:30.455049 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.455017    2562 generic.go:356] "Generic (PLEG): container finished" podID="ed2eded8-c029-4b25-9529-4432cb216130" containerID="ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b" exitCode=0
Jul 16 21:59:30.455251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.455066    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cdc9548dd-5gs7h"
Jul 16 21:59:30.455251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.455107    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-5gs7h" event={"ID":"ed2eded8-c029-4b25-9529-4432cb216130","Type":"ContainerDied","Data":"ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b"}
Jul 16 21:59:30.455251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.455165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cdc9548dd-5gs7h" event={"ID":"ed2eded8-c029-4b25-9529-4432cb216130","Type":"ContainerDied","Data":"2a3fd120b0e37784e5ef7863a087b343f8f25055b046156a414ae686336fa730"}
Jul 16 21:59:30.455251 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.455194    2562 scope.go:117] "RemoveContainer" containerID="ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b"
Jul 16 21:59:30.463869 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.463853    2562 scope.go:117] "RemoveContainer" containerID="ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b"
Jul 16 21:59:30.464114 ip-10-0-139-156 kubenswrapper[2562]: E0716 21:59:30.464098    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b\": container with ID starting with ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b not found: ID does not exist" containerID="ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b"
Jul 16 21:59:30.464152 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.464124    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b"} err="failed to get container status \"ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b\": rpc error: code = NotFound desc = could not find container \"ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b\": container with ID starting with ba7fc16eb3d8a954ed790646a87a82b63f31993d94334e0b1bcd5fb3875c2c2b not found: ID does not exist"
Jul 16 21:59:30.475537 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.475511    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-5gs7h"]
Jul 16 21:59:30.479253 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:30.479231    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cdc9548dd-5gs7h"]
Jul 16 21:59:31.750398 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:31.750364    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2eded8-c029-4b25-9529-4432cb216130" path="/var/lib/kubelet/pods/ed2eded8-c029-4b25-9529-4432cb216130/volumes"
Jul 16 21:59:49.724993 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:49.724947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:49.826125 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:49.826092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:50.239757 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.239725    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:50.338095 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.338065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:50.800526 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.800492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:50.898674 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.898641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 21:59:50.911743 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.911714    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"]
Jul 16 21:59:50.917100 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.917058    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2eded8-c029-4b25-9529-4432cb216130" containerName="manager"
Jul 16 21:59:50.917216 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.917196    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2eded8-c029-4b25-9529-4432cb216130" containerName="manager"
Jul 16 21:59:50.917397 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.917385    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2eded8-c029-4b25-9529-4432cb216130" containerName="manager"
Jul 16 21:59:50.920943 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.920920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:50.923075 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.923050    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"]
Jul 16 21:59:50.923816 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.923639    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 21:59:50.923816 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:50.923639    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dj5fj\""
Jul 16 21:59:51.057103 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.057024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7hh9\" (UniqueName: \"kubernetes.io/projected/a9cf7475-5dcb-40af-8636-1882c45278f3-kube-api-access-n7hh9\") pod \"payload-pre-processing-7677b774bb-r8vkg\" (UID: \"a9cf7475-5dcb-40af-8636-1882c45278f3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.057103 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.057080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cf7475-5dcb-40af-8636-1882c45278f3-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-r8vkg\" (UID: \"a9cf7475-5dcb-40af-8636-1882c45278f3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.158449 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.158409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7hh9\" (UniqueName: \"kubernetes.io/projected/a9cf7475-5dcb-40af-8636-1882c45278f3-kube-api-access-n7hh9\") pod \"payload-pre-processing-7677b774bb-r8vkg\" (UID: \"a9cf7475-5dcb-40af-8636-1882c45278f3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.158659 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.158474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cf7475-5dcb-40af-8636-1882c45278f3-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-r8vkg\" (UID: \"a9cf7475-5dcb-40af-8636-1882c45278f3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.159216 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.159190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cf7475-5dcb-40af-8636-1882c45278f3-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-r8vkg\" (UID: \"a9cf7475-5dcb-40af-8636-1882c45278f3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.166625 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.166597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7hh9\" (UniqueName: \"kubernetes.io/projected/a9cf7475-5dcb-40af-8636-1882c45278f3-kube-api-access-n7hh9\") pod \"payload-pre-processing-7677b774bb-r8vkg\" (UID: \"a9cf7475-5dcb-40af-8636-1882c45278f3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.233208 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.233172    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:51.372961 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.372888    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"]
Jul 16 21:59:51.375532 ip-10-0-139-156 kubenswrapper[2562]: W0716 21:59:51.375506    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cf7475_5dcb_40af_8636_1882c45278f3.slice/crio-0944dedb3d9ae8c222c672b7bd4d3cf256a3db149a3a2b3445f8171576f7f129 WatchSource:0}: Error finding container 0944dedb3d9ae8c222c672b7bd4d3cf256a3db149a3a2b3445f8171576f7f129: Status 404 returned error can't find the container with id 0944dedb3d9ae8c222c672b7bd4d3cf256a3db149a3a2b3445f8171576f7f129
Jul 16 21:59:51.543773 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:51.543741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg" event={"ID":"a9cf7475-5dcb-40af-8636-1882c45278f3","Type":"ContainerStarted","Data":"0944dedb3d9ae8c222c672b7bd4d3cf256a3db149a3a2b3445f8171576f7f129"}
Jul 16 21:59:54.557800 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:54.557762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg" event={"ID":"a9cf7475-5dcb-40af-8636-1882c45278f3","Type":"ContainerStarted","Data":"728b501fc48935dde0613a5662bdfd3f0e31d30b6654e3cc9373abf09efa808e"}
Jul 16 21:59:54.558196 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:54.557880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 21:59:54.576031 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:54.575981    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg" podStartSLOduration=2.296608031 podStartE2EDuration="4.575966136s" totalImagesPullingTime="2.279358105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:54.575456249 +0000 UTC m=+777.357354099" watchObservedRunningTime="2026-07-16 21:59:54.575966136 +0000 UTC m=+777.357863984"
Jul 16 21:59:59.967399 ip-10-0-139-156 kubenswrapper[2562]: I0716 21:59:59.967366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:00.064007 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.063977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:00.160511 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.160474    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"]
Jul 16 22:00:00.164965 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.164945    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"
Jul 16 22:00:00.168855 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.168835    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:00:00.174040 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.174023    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"
Jul 16 22:00:00.184085 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.184059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"]
Jul 16 22:00:00.343116 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.343094    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"]
Jul 16 22:00:00.345268 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:00:00.345241    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc81bf62_2a28_45cf_b7c6_7821758b4421.slice/crio-9d76dc9f2c8d58b8165775528e881a2150408d24705e594fe468e0e18b8af090 WatchSource:0}: Error finding container 9d76dc9f2c8d58b8165775528e881a2150408d24705e594fe468e0e18b8af090: Status 404 returned error can't find the container with id 9d76dc9f2c8d58b8165775528e881a2150408d24705e594fe468e0e18b8af090
Jul 16 22:00:00.586666 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:00.586576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7" event={"ID":"dc81bf62-2a28-45cf-b7c6-7821758b4421","Type":"ContainerStarted","Data":"9d76dc9f2c8d58b8165775528e881a2150408d24705e594fe468e0e18b8af090"}
Jul 16 22:00:04.604433 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:04.604347    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7" event={"ID":"dc81bf62-2a28-45cf-b7c6-7821758b4421","Type":"ContainerStarted","Data":"eb1c040ef0a2ceea2ab68e3e68296b24eee8c9ecbf0b8deb654bcbe049356246"}
Jul 16 22:00:04.618871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:04.618821    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7" podStartSLOduration=1.011047571 podStartE2EDuration="4.618804989s" totalImagesPullingTime="3.607757418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:04.618020877 +0000 UTC m=+787.399918726" watchObservedRunningTime="2026-07-16 22:00:04.618804989 +0000 UTC m=+787.400702837"
Jul 16 22:00:05.564935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:05.564905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-r8vkg"
Jul 16 22:00:05.610799 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:05.610765    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc81bf62-2a28-45cf-b7c6-7821758b4421" containerID="eb1c040ef0a2ceea2ab68e3e68296b24eee8c9ecbf0b8deb654bcbe049356246" exitCode=0
Jul 16 22:00:05.611161 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:05.610861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7" event={"ID":"dc81bf62-2a28-45cf-b7c6-7821758b4421","Type":"ContainerDied","Data":"eb1c040ef0a2ceea2ab68e3e68296b24eee8c9ecbf0b8deb654bcbe049356246"}
Jul 16 22:00:06.749946 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:06.749921    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"
Jul 16 22:00:07.621032 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:07.621004    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"
Jul 16 22:00:07.621204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:07.621031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7" event={"ID":"dc81bf62-2a28-45cf-b7c6-7821758b4421","Type":"ContainerDied","Data":"9d76dc9f2c8d58b8165775528e881a2150408d24705e594fe468e0e18b8af090"}
Jul 16 22:00:07.621204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:07.621071    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d76dc9f2c8d58b8165775528e881a2150408d24705e594fe468e0e18b8af090"
Jul 16 22:00:13.277618 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:13.277582    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:24.906353 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.906318    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"]
Jul 16 22:00:24.906746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.906640    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc81bf62-2a28-45cf-b7c6-7821758b4421" containerName="cleanup"
Jul 16 22:00:24.906746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.906652    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc81bf62-2a28-45cf-b7c6-7821758b4421" containerName="cleanup"
Jul 16 22:00:24.906746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.906707    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc81bf62-2a28-45cf-b7c6-7821758b4421" containerName="cleanup"
Jul 16 22:00:24.910520 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.910491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:24.914810 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.914780    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 16 22:00:24.914810 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.914802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rr6sx\""
Jul 16 22:00:24.914810 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.914811    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 22:00:24.915053 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.914802    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 22:00:24.919301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.919276    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"]
Jul 16 22:00:24.953359 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.953326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66383d6-fd59-46ce-a7b3-e257c01cca2e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:24.953589 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.953369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhn8c\" (UniqueName: \"kubernetes.io/projected/f66383d6-fd59-46ce-a7b3-e257c01cca2e-kube-api-access-mhn8c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:24.953589 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.953396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:24.953589 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.953414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:24.953589 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.953493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:24.953768 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:24.953596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.054282 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.054282 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.054595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.054595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66383d6-fd59-46ce-a7b3-e257c01cca2e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.054595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhn8c\" (UniqueName: \"kubernetes.io/projected/f66383d6-fd59-46ce-a7b3-e257c01cca2e-kube-api-access-mhn8c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.054595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.055765 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.055765 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.055765 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.054820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.056988 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.056967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66383d6-fd59-46ce-a7b3-e257c01cca2e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.057165 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.057144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66383d6-fd59-46ce-a7b3-e257c01cca2e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.063949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.063931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhn8c\" (UniqueName: \"kubernetes.io/projected/f66383d6-fd59-46ce-a7b3-e257c01cca2e-kube-api-access-mhn8c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tllmv\" (UID: \"f66383d6-fd59-46ce-a7b3-e257c01cca2e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.221969 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.221885    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:25.358390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.358364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"]
Jul 16 22:00:25.361054 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:00:25.361029    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66383d6_fd59_46ce_a7b3_e257c01cca2e.slice/crio-22e7acae02ef533aaf0fb460e95f0d8576a165bf03ec4bde779e0db3f3124e5f WatchSource:0}: Error finding container 22e7acae02ef533aaf0fb460e95f0d8576a165bf03ec4bde779e0db3f3124e5f: Status 404 returned error can't find the container with id 22e7acae02ef533aaf0fb460e95f0d8576a165bf03ec4bde779e0db3f3124e5f
Jul 16 22:00:25.699700 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:25.699638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv" event={"ID":"f66383d6-fd59-46ce-a7b3-e257c01cca2e","Type":"ContainerStarted","Data":"22e7acae02ef533aaf0fb460e95f0d8576a165bf03ec4bde779e0db3f3124e5f"}
Jul 16 22:00:26.329109 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:26.329078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:26.705933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:26.705896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv" event={"ID":"f66383d6-fd59-46ce-a7b3-e257c01cca2e","Type":"ContainerStarted","Data":"98794d9445d73c1fd7cc2d7d9283ee144c337f1ace1baefd188bb914f089be24"}
Jul 16 22:00:26.724084 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:26.724037    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv" podStartSLOduration=1.5733811370000002 podStartE2EDuration="2.724019053s" totalImagesPullingTime="1.150637916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:26.722394645 +0000 UTC m=+809.504292492" watchObservedRunningTime="2026-07-16 22:00:26.724019053 +0000 UTC m=+809.505916901"
Jul 16 22:00:35.222175 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:35.222130    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:35.222770 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:35.222192    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:35.234755 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:35.234730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:35.757876 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:35.757849    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tllmv"
Jul 16 22:00:38.631111 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:38.631082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:52.909067 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.909035    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"]
Jul 16 22:00:52.912271 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.912250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:52.914685 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.914664    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 22:00:52.921409 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.921384    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"]
Jul 16 22:00:52.999836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.999802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.000025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.999854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.000025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.999937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.000025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:52.999984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.000199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.000039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3483aa9e-7b82-4422-aa94-c702e7eb32d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.000199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.000062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57kvj\" (UniqueName: \"kubernetes.io/projected/3483aa9e-7b82-4422-aa94-c702e7eb32d2-kube-api-access-57kvj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101403 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57kvj\" (UniqueName: \"kubernetes.io/projected/3483aa9e-7b82-4422-aa94-c702e7eb32d2-kube-api-access-57kvj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101612 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101612 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101612 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101612 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101813 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3483aa9e-7b82-4422-aa94-c702e7eb32d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101956 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.101992 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.101965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.102025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.102007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.104053 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.104028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3483aa9e-7b82-4422-aa94-c702e7eb32d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.104284 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.104267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3483aa9e-7b82-4422-aa94-c702e7eb32d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.110249 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.110220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57kvj\" (UniqueName: \"kubernetes.io/projected/3483aa9e-7b82-4422-aa94-c702e7eb32d2-kube-api-access-57kvj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt\" (UID: \"3483aa9e-7b82-4422-aa94-c702e7eb32d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.121214 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.121192    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:53.223467 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.223401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:00:53.376899 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.376874    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"]
Jul 16 22:00:53.379244 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:00:53.379217    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3483aa9e_7b82_4422_aa94_c702e7eb32d2.slice/crio-927ca3be8da55d4b43b0988becbc5ab177b941eecb12c6b370b1854548838d44 WatchSource:0}: Error finding container 927ca3be8da55d4b43b0988becbc5ab177b941eecb12c6b370b1854548838d44: Status 404 returned error can't find the container with id 927ca3be8da55d4b43b0988becbc5ab177b941eecb12c6b370b1854548838d44
Jul 16 22:00:53.819365 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.819331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt" event={"ID":"3483aa9e-7b82-4422-aa94-c702e7eb32d2","Type":"ContainerStarted","Data":"4b33c15fbd61b3736246e24ac487025fbe266db8c3ee3d4972043a106ed3b9b8"}
Jul 16 22:00:53.819602 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.819395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt" event={"ID":"3483aa9e-7b82-4422-aa94-c702e7eb32d2","Type":"ContainerStarted","Data":"927ca3be8da55d4b43b0988becbc5ab177b941eecb12c6b370b1854548838d44"}
Jul 16 22:00:53.837179 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:00:53.837131    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt" podStartSLOduration=1.62824474 podStartE2EDuration="1.837115423s" totalImagesPullingTime="208.870683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:53.836174457 +0000 UTC m=+836.618072302" watchObservedRunningTime="2026-07-16 22:00:53.837115423 +0000 UTC m=+836.619013271"
Jul 16 22:01:00.402614 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.402586    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"]
Jul 16 22:01:00.405944 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.405923    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.408436 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.408417    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 16 22:01:00.419904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.419882    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"]
Jul 16 22:01:00.452601 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.452575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:00.464948 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.464925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.465032 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.464962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnb8r\" (UniqueName: \"kubernetes.io/projected/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-kube-api-access-hnb8r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.465032 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.464983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.465107 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.465052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.465107 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.465089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.465107 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.465104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565456 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565456 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565697 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565697 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnb8r\" (UniqueName: \"kubernetes.io/projected/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-kube-api-access-hnb8r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565697 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565697 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.565950 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.566014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.565952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.566014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.566006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.567888 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.567867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.568119 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.568102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.574748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.574725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnb8r\" (UniqueName: \"kubernetes.io/projected/e19e3f71-e690-4d4f-aa38-6d17e0f9f592-kube-api-access-hnb8r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6jjvc\" (UID: \"e19e3f71-e690-4d4f-aa38-6d17e0f9f592\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.716851 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.716762    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:00.850005 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:00.849980    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"]
Jul 16 22:01:00.852421 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:01:00.852395    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19e3f71_e690_4d4f_aa38_6d17e0f9f592.slice/crio-0e77e0a21fa3fc9610fc1673f9d2bf864109868ddb9c758a9a9933f9a7dfd4b8 WatchSource:0}: Error finding container 0e77e0a21fa3fc9610fc1673f9d2bf864109868ddb9c758a9a9933f9a7dfd4b8: Status 404 returned error can't find the container with id 0e77e0a21fa3fc9610fc1673f9d2bf864109868ddb9c758a9a9933f9a7dfd4b8
Jul 16 22:01:01.855917 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:01.855868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc" event={"ID":"e19e3f71-e690-4d4f-aa38-6d17e0f9f592","Type":"ContainerStarted","Data":"0e77e0a21fa3fc9610fc1673f9d2bf864109868ddb9c758a9a9933f9a7dfd4b8"}
Jul 16 22:01:01.961896 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:01.961857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:02.067508 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:02.067475    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:02.862212 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:02.862169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc" event={"ID":"e19e3f71-e690-4d4f-aa38-6d17e0f9f592","Type":"ContainerStarted","Data":"2893f9fb7ed4c70996c5a09695a69cb5d485c8f8cb87223348766ed3e86713b9"}
Jul 16 22:01:02.883257 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:02.883200    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc" podStartSLOduration=1.042014609 podStartE2EDuration="2.883182614s" totalImagesPullingTime="1.841168005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:02.881608206 +0000 UTC m=+845.663506044" watchObservedRunningTime="2026-07-16 22:01:02.883182614 +0000 UTC m=+845.665080462"
Jul 16 22:01:03.224282 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:03.224241    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:01:03.224464 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:03.224329    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:01:03.237193 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:03.237169    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:01:03.878336 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:03.878307    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt"
Jul 16 22:01:09.603806 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:09.603770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:09.729399 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:09.729363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:10.717297 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:10.717261    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:10.717820 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:10.717370    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:10.729739 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:10.729713    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:01:10.912437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:01:10.912400    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6jjvc"
Jul 16 22:02:31.448788 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:02:31.448755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:34.720418 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:02:34.720383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:04:20.394836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.394802    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-95466d4fd-pcbh7"]
Jul 16 22:04:20.398098 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.398078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.403697 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.403678    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sckmc\""
Jul 16 22:04:20.404952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.404932    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 22:04:20.405039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.404942    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 22:04:20.411858 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.411840    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-95466d4fd-pcbh7"]
Jul 16 22:04:20.492981 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.492947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zh4m\" (UniqueName: \"kubernetes.io/projected/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-kube-api-access-2zh4m\") pod \"maas-controller-95466d4fd-pcbh7\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") " pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.493165 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.493011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-webhook-cert\") pod \"maas-controller-95466d4fd-pcbh7\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") " pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.593972 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.593936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-webhook-cert\") pod \"maas-controller-95466d4fd-pcbh7\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") " pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.594132 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.594002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh4m\" (UniqueName: \"kubernetes.io/projected/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-kube-api-access-2zh4m\") pod \"maas-controller-95466d4fd-pcbh7\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") " pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.596595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.596542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-webhook-cert\") pod \"maas-controller-95466d4fd-pcbh7\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") " pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.604182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.604159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh4m\" (UniqueName: \"kubernetes.io/projected/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-kube-api-access-2zh4m\") pod \"maas-controller-95466d4fd-pcbh7\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") " pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.707874 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.707840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:20.854056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.854032    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-95466d4fd-pcbh7"]
Jul 16 22:04:20.856652 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:04:20.856619    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f22313_09c0_4ed2_91a6_35cf5fc74bdf.slice/crio-9278c48e3c5990e4e202131493f012c080b790ab2683163bea22e6b0ff80cce6 WatchSource:0}: Error finding container 9278c48e3c5990e4e202131493f012c080b790ab2683163bea22e6b0ff80cce6: Status 404 returned error can't find the container with id 9278c48e3c5990e4e202131493f012c080b790ab2683163bea22e6b0ff80cce6
Jul 16 22:04:20.858254 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:20.858233    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:04:21.750954 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:21.750919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-95466d4fd-pcbh7" event={"ID":"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf","Type":"ContainerStarted","Data":"e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de"}
Jul 16 22:04:21.751333 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:21.750970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-95466d4fd-pcbh7" event={"ID":"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf","Type":"ContainerStarted","Data":"9278c48e3c5990e4e202131493f012c080b790ab2683163bea22e6b0ff80cce6"}
Jul 16 22:04:21.751333 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:21.750988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:21.795837 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:21.795788    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-95466d4fd-pcbh7" podStartSLOduration=1.328980467 podStartE2EDuration="1.795774638s" totalImagesPullingTime="466.794171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:04:21.793684371 +0000 UTC m=+1044.575582220" watchObservedRunningTime="2026-07-16 22:04:21.795774638 +0000 UTC m=+1044.577672485"
Jul 16 22:04:33.760838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:33.760806    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:38.334391 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.334335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:04:38.409740 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.409703    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-95466d4fd-pcbh7"]
Jul 16 22:04:38.410002 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.409976    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-95466d4fd-pcbh7" podUID="c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" containerName="manager" containerID="cri-o://e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de" gracePeriod=30
Jul 16 22:04:38.666006 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.665977    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:38.747025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.746994    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-webhook-cert\") pod \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") "
Jul 16 22:04:38.747182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.747060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zh4m\" (UniqueName: \"kubernetes.io/projected/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-kube-api-access-2zh4m\") pod \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\" (UID: \"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf\") "
Jul 16 22:04:38.749116 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.749090    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" (UID: "c7f22313-09c0-4ed2-91a6-35cf5fc74bdf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:04:38.749237 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.749214    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-kube-api-access-2zh4m" (OuterVolumeSpecName: "kube-api-access-2zh4m") pod "c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" (UID: "c7f22313-09c0-4ed2-91a6-35cf5fc74bdf"). InnerVolumeSpecName "kube-api-access-2zh4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:04:38.824636 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.824602    2562 generic.go:356] "Generic (PLEG): container finished" podID="c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" containerID="e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de" exitCode=0
Jul 16 22:04:38.824793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.824641    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-95466d4fd-pcbh7"
Jul 16 22:04:38.824793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.824688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-95466d4fd-pcbh7" event={"ID":"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf","Type":"ContainerDied","Data":"e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de"}
Jul 16 22:04:38.824793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.824735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-95466d4fd-pcbh7" event={"ID":"c7f22313-09c0-4ed2-91a6-35cf5fc74bdf","Type":"ContainerDied","Data":"9278c48e3c5990e4e202131493f012c080b790ab2683163bea22e6b0ff80cce6"}
Jul 16 22:04:38.824793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.824760    2562 scope.go:117] "RemoveContainer" containerID="e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de"
Jul 16 22:04:38.834278 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.834257    2562 scope.go:117] "RemoveContainer" containerID="e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de"
Jul 16 22:04:38.834535 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:04:38.834515    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de\": container with ID starting with e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de not found: ID does not exist" containerID="e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de"
Jul 16 22:04:38.834703 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.834543    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de"} err="failed to get container status \"e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de\": rpc error: code = NotFound desc = could not find container \"e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de\": container with ID starting with e868f5e8b1ea45574e68207ff1fde20857d8feffbda82b753fdcc2ba226934de not found: ID does not exist"
Jul 16 22:04:38.845450 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.845427    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-95466d4fd-pcbh7"]
Jul 16 22:04:38.848116 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.848100    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-webhook-cert\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:04:38.848176 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.848119    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zh4m\" (UniqueName: \"kubernetes.io/projected/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf-kube-api-access-2zh4m\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:04:38.852339 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:38.852321    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-95466d4fd-pcbh7"]
Jul 16 22:04:39.523792 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.523751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:04:39.639531 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.639496    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-95466d4fd-mdsdq"]
Jul 16 22:04:39.639923 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.639905    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" containerName="manager"
Jul 16 22:04:39.640008 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.639926    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" containerName="manager"
Jul 16 22:04:39.640060 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.640019    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" containerName="manager"
Jul 16 22:04:39.644207 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.644184    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.647752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.647728    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sckmc\""
Jul 16 22:04:39.647752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.647730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 22:04:39.647962 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.647768    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 22:04:39.652238 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.652215    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-95466d4fd-mdsdq"]
Jul 16 22:04:39.750494 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.750452    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f22313-09c0-4ed2-91a6-35cf5fc74bdf" path="/var/lib/kubelet/pods/c7f22313-09c0-4ed2-91a6-35cf5fc74bdf/volumes"
Jul 16 22:04:39.756009 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.755988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bk2g\" (UniqueName: \"kubernetes.io/projected/c5f395d0-76cf-47bf-a1c4-1e15a9ed061c-kube-api-access-4bk2g\") pod \"maas-controller-95466d4fd-mdsdq\" (UID: \"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c\") " pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.756118 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.756035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5f395d0-76cf-47bf-a1c4-1e15a9ed061c-webhook-cert\") pod \"maas-controller-95466d4fd-mdsdq\" (UID: \"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c\") " pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.856564 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.856458    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bk2g\" (UniqueName: \"kubernetes.io/projected/c5f395d0-76cf-47bf-a1c4-1e15a9ed061c-kube-api-access-4bk2g\") pod \"maas-controller-95466d4fd-mdsdq\" (UID: \"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c\") " pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.856564 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.856508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5f395d0-76cf-47bf-a1c4-1e15a9ed061c-webhook-cert\") pod \"maas-controller-95466d4fd-mdsdq\" (UID: \"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c\") " pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.859177 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.859152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5f395d0-76cf-47bf-a1c4-1e15a9ed061c-webhook-cert\") pod \"maas-controller-95466d4fd-mdsdq\" (UID: \"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c\") " pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.864154 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.864133    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bk2g\" (UniqueName: \"kubernetes.io/projected/c5f395d0-76cf-47bf-a1c4-1e15a9ed061c-kube-api-access-4bk2g\") pod \"maas-controller-95466d4fd-mdsdq\" (UID: \"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c\") " pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:39.956268 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:39.956235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:40.088173 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:40.088149    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-95466d4fd-mdsdq"]
Jul 16 22:04:40.090479 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:04:40.090449    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f395d0_76cf_47bf_a1c4_1e15a9ed061c.slice/crio-fb463a647935e5e377190d7e1b32776dfb9e27bdbe18b68858de4136705e0b21 WatchSource:0}: Error finding container fb463a647935e5e377190d7e1b32776dfb9e27bdbe18b68858de4136705e0b21: Status 404 returned error can't find the container with id fb463a647935e5e377190d7e1b32776dfb9e27bdbe18b68858de4136705e0b21
Jul 16 22:04:40.835948 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:40.835867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-95466d4fd-mdsdq" event={"ID":"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c","Type":"ContainerStarted","Data":"71a23cf959c66e451c44361e9757474f20b706bfc0a888823fe858838c60f40b"}
Jul 16 22:04:40.835948 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:40.835920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-95466d4fd-mdsdq" event={"ID":"c5f395d0-76cf-47bf-a1c4-1e15a9ed061c","Type":"ContainerStarted","Data":"fb463a647935e5e377190d7e1b32776dfb9e27bdbe18b68858de4136705e0b21"}
Jul 16 22:04:40.836358 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:40.835988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:04:40.856391 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:40.856340    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-95466d4fd-mdsdq" podStartSLOduration=1.543799919 podStartE2EDuration="1.856321414s" totalImagesPullingTime="312.521495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:04:40.85466262 +0000 UTC m=+1063.636560471" watchObservedRunningTime="2026-07-16 22:04:40.856321414 +0000 UTC m=+1063.638219311"
Jul 16 22:04:51.844376 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:04:51.844295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-95466d4fd-mdsdq"
Jul 16 22:05:20.318331 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:20.318288    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:20.422921 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:20.422883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:20.518450 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:20.518411    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:26.480991 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:26.480947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:26.583996 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:26.583954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:26.680911 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:26.680880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:26.841126 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:26.841039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:33.272741 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:33.272695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:33.435608 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:05:33.435568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:06:02.936233 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:06:02.936192    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:06:10.972735 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:06:10.972696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:07:56.193819 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:07:56.193708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:07:58.541091 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:07:58.541054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:08:50.820976 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:08:50.820939    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:08:50.921861 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:08:50.921806    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:09:29.795163 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:29.795130    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"]
Jul 16 22:09:29.795723 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:29.795324    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26" podUID="9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" containerName="manager" containerID="cri-o://151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9" gracePeriod=10
Jul 16 22:09:32.736325 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.736299    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 22:09:32.830172 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.830072    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-extensions-socket-volume\") pod \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") "
Jul 16 22:09:32.830326 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.830177    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdrmt\" (UniqueName: \"kubernetes.io/projected/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-kube-api-access-gdrmt\") pod \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\" (UID: \"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7\") "
Jul 16 22:09:32.830486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.830456    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" (UID: "9279f9e8-d36b-45c3-a2b9-36ccc6a09df7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:32.832284 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.832263    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-kube-api-access-gdrmt" (OuterVolumeSpecName: "kube-api-access-gdrmt") pod "9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" (UID: "9279f9e8-d36b-45c3-a2b9-36ccc6a09df7"). InnerVolumeSpecName "kube-api-access-gdrmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:09:32.931558 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.931524    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-extensions-socket-volume\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:09:32.931719 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:32.931575    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdrmt\" (UniqueName: \"kubernetes.io/projected/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7-kube-api-access-gdrmt\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:09:33.144368 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.144278    2562 generic.go:356] "Generic (PLEG): container finished" podID="9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" containerID="151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9" exitCode=0
Jul 16 22:09:33.144368 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.144330    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"
Jul 16 22:09:33.144368 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.144339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26" event={"ID":"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7","Type":"ContainerDied","Data":"151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9"}
Jul 16 22:09:33.144639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.144379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26" event={"ID":"9279f9e8-d36b-45c3-a2b9-36ccc6a09df7","Type":"ContainerDied","Data":"49184aee3a0bb5dea125ffa432f11af237e3fe46b0652692ef77f9718fc7eadb"}
Jul 16 22:09:33.144639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.144399    2562 scope.go:117] "RemoveContainer" containerID="151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9"
Jul 16 22:09:33.154241 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.154223    2562 scope.go:117] "RemoveContainer" containerID="151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9"
Jul 16 22:09:33.154516 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:09:33.154496    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9\": container with ID starting with 151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9 not found: ID does not exist" containerID="151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9"
Jul 16 22:09:33.154597 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.154526    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9"} err="failed to get container status \"151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9\": rpc error: code = NotFound desc = could not find container \"151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9\": container with ID starting with 151ce8436544ce1766fa410c924b2a5d8af5533272559a6fb46a8d227e2681b9 not found: ID does not exist"
Jul 16 22:09:33.169308 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.169282    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"]
Jul 16 22:09:33.173408 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.173387    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wjw26"]
Jul 16 22:09:33.750732 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:09:33.750700    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" path="/var/lib/kubelet/pods/9279f9e8-d36b-45c3-a2b9-36ccc6a09df7/volumes"
Jul 16 22:10:30.988640 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:30.988605    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"]
Jul 16 22:10:30.989137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:30.988921    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" containerName="manager"
Jul 16 22:10:30.989137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:30.988932    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" containerName="manager"
Jul 16 22:10:30.989137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:30.988991    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9279f9e8-d36b-45c3-a2b9-36ccc6a09df7" containerName="manager"
Jul 16 22:10:30.991911 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:30.991892    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:30.994590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:30.994569    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ld9cr\""
Jul 16 22:10:31.009078 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.009049    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"]
Jul 16 22:10:31.123577 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.123518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvlc\" (UniqueName: \"kubernetes.io/projected/a0b18074-6556-433d-bd18-4dfb8a52688b-kube-api-access-glvlc\") pod \"kuadrant-operator-controller-manager-86878c769f-lwplb\" (UID: \"a0b18074-6556-433d-bd18-4dfb8a52688b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.123747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.123581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b18074-6556-433d-bd18-4dfb8a52688b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lwplb\" (UID: \"a0b18074-6556-433d-bd18-4dfb8a52688b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.224635 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.224598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glvlc\" (UniqueName: \"kubernetes.io/projected/a0b18074-6556-433d-bd18-4dfb8a52688b-kube-api-access-glvlc\") pod \"kuadrant-operator-controller-manager-86878c769f-lwplb\" (UID: \"a0b18074-6556-433d-bd18-4dfb8a52688b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.224804 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.224640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b18074-6556-433d-bd18-4dfb8a52688b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lwplb\" (UID: \"a0b18074-6556-433d-bd18-4dfb8a52688b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.225014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.224998    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b18074-6556-433d-bd18-4dfb8a52688b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lwplb\" (UID: \"a0b18074-6556-433d-bd18-4dfb8a52688b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.235539 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.235515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvlc\" (UniqueName: \"kubernetes.io/projected/a0b18074-6556-433d-bd18-4dfb8a52688b-kube-api-access-glvlc\") pod \"kuadrant-operator-controller-manager-86878c769f-lwplb\" (UID: \"a0b18074-6556-433d-bd18-4dfb8a52688b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.302318 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.302226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:31.454831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:31.454808    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"]
Jul 16 22:10:31.458038 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:10:31.458005    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b18074_6556_433d_bd18_4dfb8a52688b.slice/crio-ad11297d0545fe1aa73be22fe72722490f808997dec660e038a7cb7065afb9a7 WatchSource:0}: Error finding container ad11297d0545fe1aa73be22fe72722490f808997dec660e038a7cb7065afb9a7: Status 404 returned error can't find the container with id ad11297d0545fe1aa73be22fe72722490f808997dec660e038a7cb7065afb9a7
Jul 16 22:10:32.396890 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:32.396850    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb" event={"ID":"a0b18074-6556-433d-bd18-4dfb8a52688b","Type":"ContainerStarted","Data":"b29c2fdd030bfd7a7616f30f18249ae7716208f0e8bcd8ccb0fad716d34d2d9d"}
Jul 16 22:10:32.397361 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:32.396908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb" event={"ID":"a0b18074-6556-433d-bd18-4dfb8a52688b","Type":"ContainerStarted","Data":"ad11297d0545fe1aa73be22fe72722490f808997dec660e038a7cb7065afb9a7"}
Jul 16 22:10:32.397361 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:32.396946    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:10:32.419241 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:32.419197    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb" podStartSLOduration=2.419182835 podStartE2EDuration="2.419182835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:10:32.416649261 +0000 UTC m=+1415.198547110" watchObservedRunningTime="2026-07-16 22:10:32.419182835 +0000 UTC m=+1415.201080686"
Jul 16 22:10:43.402950 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:10:43.402915    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lwplb"
Jul 16 22:15:41.183015 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:15:41.182979    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:29.876715 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:29.876672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:35.223657 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:35.223611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:43.583279 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:43.583236    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:48.772490 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:48.772395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:51.798073 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:51.798026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:51.895496 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:51.895457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:52.267700 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:52.267664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:58.273129 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:58.273085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:59.581885 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:59.581842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:59.708626 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:59.708593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:16:59.927791 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:16:59.927754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:00.034816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:00.034783    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:01.079686 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.079645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:01.291372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.291334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:01.328733 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.328692    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"]
Jul 16 22:17:01.332137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.332078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.335423 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.335401    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:17:01.336030 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.336011    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-af8fc2dc-serving-cert\""
Jul 16 22:17:01.336115 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.336014    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:17:01.336159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.336014    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:17:01.349017 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.348991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"]
Jul 16 22:17:01.394562 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.394528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-openshift-service-ca\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.394742 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.394588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.394742 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.394686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzs2s\" (UniqueName: \"kubernetes.io/projected/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-kube-api-access-pzs2s\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.415529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.415494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:01.495877 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.495844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs2s\" (UniqueName: \"kubernetes.io/projected/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-kube-api-access-pzs2s\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.496044 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.495911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-openshift-service-ca\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.496044 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.495936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.496044 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:01.496029    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-serving-cert: secret "maas-api-e2e-ait-af8fc2dc-serving-cert" not found
Jul 16 22:17:01.496158 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:01.496086    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls podName:3bfd0f85-b8fa-483d-9bf0-e5a13b96745b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:01.996068885 +0000 UTC m=+1804.777966712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls") pod "maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" (UID: "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b") : secret "maas-api-e2e-ait-af8fc2dc-serving-cert" not found
Jul 16 22:17:01.496639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.496621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-openshift-service-ca\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.510645 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.510615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs2s\" (UniqueName: \"kubernetes.io/projected/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-kube-api-access-pzs2s\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:01.893205 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:01.893164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:02.000306 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.000269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:02.002916 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.002887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls\") pod \"maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:02.243303 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.243270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:02.406141 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.406107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:02.436335 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.436309    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"]
Jul 16 22:17:02.440276 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:17:02.439569    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfd0f85_b8fa_483d_9bf0_e5a13b96745b.slice/crio-22e709b863a28c39d623e28e622579b47a5de3dd112d5bd3411df004a15a95dd WatchSource:0}: Error finding container 22e709b863a28c39d623e28e622579b47a5de3dd112d5bd3411df004a15a95dd: Status 404 returned error can't find the container with id 22e709b863a28c39d623e28e622579b47a5de3dd112d5bd3411df004a15a95dd
Jul 16 22:17:02.441191 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.441164    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:17:02.509847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:02.509757    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:03.133047 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:03.133006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerStarted","Data":"22e709b863a28c39d623e28e622579b47a5de3dd112d5bd3411df004a15a95dd"}
Jul 16 22:17:03.767252 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:03.767205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:17:05.144839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:05.144803    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerID="706c3124af13a744f3ce11d827ae0ef33da5c77a5b0021231cbd10ac49da474b" exitCode=1
Jul 16 22:17:05.145316 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:05.144854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerDied","Data":"706c3124af13a744f3ce11d827ae0ef33da5c77a5b0021231cbd10ac49da474b"}
Jul 16 22:17:05.145316 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:05.145050    2562 scope.go:117] "RemoveContainer" containerID="706c3124af13a744f3ce11d827ae0ef33da5c77a5b0021231cbd10ac49da474b"
Jul 16 22:17:06.152113 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:06.152076    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3" exitCode=1
Jul 16 22:17:06.152501 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:06.152136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerDied","Data":"03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"}
Jul 16 22:17:06.152501 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:06.152190    2562 scope.go:117] "RemoveContainer" containerID="706c3124af13a744f3ce11d827ae0ef33da5c77a5b0021231cbd10ac49da474b"
Jul 16 22:17:06.152501 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:06.152456    2562 scope.go:117] "RemoveContainer" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"
Jul 16 22:17:06.152725 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:06.152705    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:07.160367 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:07.160339    2562 scope.go:117] "RemoveContainer" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"
Jul 16 22:17:07.160924 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:07.160543    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:07.243464 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:07.243435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:08.165127 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:08.165096    2562 scope.go:117] "RemoveContainer" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"
Jul 16 22:17:08.165506 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:08.165296    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:12.244089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:12.244056    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:12.244511 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:12.244459    2562 scope.go:117] "RemoveContainer" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"
Jul 16 22:17:12.244672 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:12.244650    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:25.745119 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:25.745089    2562 scope.go:117] "RemoveContainer" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"
Jul 16 22:17:26.247640 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:26.247590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerStarted","Data":"30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"}
Jul 16 22:17:26.247905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:26.247887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:26.269223 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:26.269174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podStartSLOduration=22.680611193 podStartE2EDuration="25.269158129s" totalImagesPullingTime="2.588546936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:26.267629676 +0000 UTC m=+1829.049527524" watchObservedRunningTime="2026-07-16 22:17:26.269158129 +0000 UTC m=+1829.051055976"
Jul 16 22:17:27.253968 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:27.253935    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9" exitCode=1
Jul 16 22:17:27.253968 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:27.253971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerDied","Data":"30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"}
Jul 16 22:17:27.254462 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:27.254011    2562 scope.go:117] "RemoveContainer" containerID="03c6ec9b8f7c4ce2b0a4cd09538558281b0ebf956c82a1c093f234b8a3f713d3"
Jul 16 22:17:27.254462 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:27.254240    2562 scope.go:117] "RemoveContainer" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"
Jul 16 22:17:27.254462 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:27.254451    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:28.260620 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:28.260585    2562 scope.go:117] "RemoveContainer" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"
Jul 16 22:17:28.261008 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:28.260764    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:32.243828 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:32.243791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:17:32.244303 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:32.244154    2562 scope.go:117] "RemoveContainer" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"
Jul 16 22:17:32.244344 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:32.244320    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:43.751346 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:43.751315    2562 scope.go:117] "RemoveContainer" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"
Jul 16 22:17:43.751913 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:17:43.751587    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:17:58.745396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:58.745364    2562 scope.go:117] "RemoveContainer" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"
Jul 16 22:17:59.400237 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:59.400204    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerStarted","Data":"c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"}
Jul 16 22:17:59.400432 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:17:59.400414    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:18:00.406008 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:00.405965    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1" exitCode=1
Jul 16 22:18:00.406508 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:00.406024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerDied","Data":"c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"}
Jul 16 22:18:00.406508 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:00.406071    2562 scope.go:117] "RemoveContainer" containerID="30d9c88e9944df9c64d99ae9463ddbb5f938d3f5372728e68eeec2a4cce3d3c9"
Jul 16 22:18:00.406508 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:00.406295    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:00.406667 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:00.406565    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:01.412068 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:01.412044    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:01.412429 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:01.412218    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:02.244014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:02.243971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:18:02.416693 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:02.416659    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:02.417183 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:02.416914    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:13.745314 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:13.745259    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:13.745748 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:13.745501    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:24.745401 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:24.745372    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:24.745882 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:24.745567    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:35.745723 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:35.745694    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:35.746103 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:35.745880    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:50.745460 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:50.745429    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:51.637288 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:51.637255    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a" exitCode=1
Jul 16 22:18:51.637451 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:51.637315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerDied","Data":"695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"}
Jul 16 22:18:51.637451 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:51.637364    2562 scope.go:117] "RemoveContainer" containerID="c4afca7cb826998262e1c2b1bebb0353c47e07536093140cb0d0fadfd640b0a1"
Jul 16 22:18:51.637732 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:51.637712    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:18:51.637934 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:51.637911    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:52.243954 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:52.243917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:18:52.644037 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:52.643965    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:18:52.644171 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:52.644141    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:18:54.400849 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:54.400814    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:18:54.401234 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:18:54.401222    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:18:54.401417 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:18:54.401393    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:19:07.750620 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:19:07.750580    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:19:07.751212 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:19:07.750901    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:19:19.751433 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:19:19.751401    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:19:19.752043 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:19:19.751627    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:19:32.745299 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:19:32.745267    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:19:32.745786 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:19:32.745447    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:19:43.745164 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:19:43.745128    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:19:43.745672 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:19:43.745324    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:19:54.745890 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:19:54.745811    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:19:54.746333 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:19:54.745989    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:20:02.472617 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:02.472578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:02.625410 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:02.625376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:06.169135 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:06.169105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:06.282521 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:06.282489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:09.746016 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:09.745983    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:20:09.746543 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:09.746168    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt_odh-ai-gateway-infra(3bfd0f85-b8fa-483d-9bf0-e5a13b96745b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"
Jul 16 22:20:11.531115 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.531071    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"]
Jul 16 22:20:11.554815 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.554781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:11.664244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.664221    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:20:11.776765 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.776690    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls\") pod \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") "
Jul 16 22:20:11.776765 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.776764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-openshift-service-ca\") pod \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") "
Jul 16 22:20:11.776982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.776821    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzs2s\" (UniqueName: \"kubernetes.io/projected/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-kube-api-access-pzs2s\") pod \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\" (UID: \"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b\") "
Jul 16 22:20:11.777173 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.777147    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" (UID: "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:11.779049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.779024    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" (UID: "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:20:11.779169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.779104    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-kube-api-access-pzs2s" (OuterVolumeSpecName: "kube-api-access-pzs2s") pod "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" (UID: "3bfd0f85-b8fa-483d-9bf0-e5a13b96745b"). InnerVolumeSpecName "kube-api-access-pzs2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:20:11.878249 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.878213    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.878249 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.878244    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.878249 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.878254    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzs2s\" (UniqueName: \"kubernetes.io/projected/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b-kube-api-access-pzs2s\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.998983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.998952    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"
Jul 16 22:20:11.999167 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.998983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt" event={"ID":"3bfd0f85-b8fa-483d-9bf0-e5a13b96745b","Type":"ContainerDied","Data":"22e709b863a28c39d623e28e622579b47a5de3dd112d5bd3411df004a15a95dd"}
Jul 16 22:20:11.999167 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:11.999047    2562 scope.go:117] "RemoveContainer" containerID="695e98d1170c6f9cc8b4637eea382c603f4a632cba0d921454a907868d8afd2a"
Jul 16 22:20:12.026719 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:12.026695    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"]
Jul 16 22:20:12.029763 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:12.029709    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af8fc2dc-6fd75c9d7f-4cvqt"]
Jul 16 22:20:12.414072 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:12.414041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:13.754510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:13.754477    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" path="/var/lib/kubelet/pods/3bfd0f85-b8fa-483d-9bf0-e5a13b96745b/volumes"
Jul 16 22:20:17.836232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:17.836205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:18.359826 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:18.359791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:18.459458 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:18.459421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:18.666852 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:18.666813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:18.771739 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:18.771706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:19.047617 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:19.047517    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:19.212198 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:19.212163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:20.152196 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.152140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:20.236609 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236576    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"]
Jul 16 22:20:20.236931 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236920    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.236974 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236933    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.236974 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236940    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.236974 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236946    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.236974 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236965    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.236974 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.236970    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.237147 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.237030    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.237147 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.237038    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.237147 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.237046    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.237147 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.237052    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.237147 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.237059    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:20:20.240922 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.240906    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.243671 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.243646    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:20:20.243865 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.243654    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:20:20.244527 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.244509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:20:20.244631 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.244542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4d06d9c2-serving-cert\""
Jul 16 22:20:20.255989 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.255967    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"]
Jul 16 22:20:20.279143 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.279116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:20.356214 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.356181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4464-5753-4097-88d0-95540b665c09-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.356392 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.356249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvz22\" (UniqueName: \"kubernetes.io/projected/8ecc4464-5753-4097-88d0-95540b665c09-kube-api-access-pvz22\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.356392 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.356337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ecc4464-5753-4097-88d0-95540b665c09-maas-api-tls\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.456966 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.456880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4464-5753-4097-88d0-95540b665c09-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.456966 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.456927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvz22\" (UniqueName: \"kubernetes.io/projected/8ecc4464-5753-4097-88d0-95540b665c09-kube-api-access-pvz22\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.456966 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.456956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ecc4464-5753-4097-88d0-95540b665c09-maas-api-tls\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.457675 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.457650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4464-5753-4097-88d0-95540b665c09-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.459652 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.459626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ecc4464-5753-4097-88d0-95540b665c09-maas-api-tls\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.466175 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.466151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvz22\" (UniqueName: \"kubernetes.io/projected/8ecc4464-5753-4097-88d0-95540b665c09-kube-api-access-pvz22\") pod \"maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.473723 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.473701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:20.551720 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.551689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:20.684983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.684952    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"]
Jul 16 22:20:20.688871 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:20:20.688814    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecc4464_5753_4097_88d0_95540b665c09.slice/crio-89357389c8c1b120a819789cc1690bd15a7b58307b52b50b605f42f498ad8f79 WatchSource:0}: Error finding container 89357389c8c1b120a819789cc1690bd15a7b58307b52b50b605f42f498ad8f79: Status 404 returned error can't find the container with id 89357389c8c1b120a819789cc1690bd15a7b58307b52b50b605f42f498ad8f79
Jul 16 22:20:20.877416 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.877331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:20.972800 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:20.972770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:21.040861 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:21.040828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" event={"ID":"8ecc4464-5753-4097-88d0-95540b665c09","Type":"ContainerStarted","Data":"89357389c8c1b120a819789cc1690bd15a7b58307b52b50b605f42f498ad8f79"}
Jul 16 22:20:22.047281 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:22.047240    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ecc4464-5753-4097-88d0-95540b665c09" containerID="6bd2c0eddf2bc9ffbce755c3c1e0290580b58765d9d0bdb2e5d9668df2f51183" exitCode=1
Jul 16 22:20:22.047777 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:22.047326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" event={"ID":"8ecc4464-5753-4097-88d0-95540b665c09","Type":"ContainerDied","Data":"6bd2c0eddf2bc9ffbce755c3c1e0290580b58765d9d0bdb2e5d9668df2f51183"}
Jul 16 22:20:22.047777 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:22.047621    2562 scope.go:117] "RemoveContainer" containerID="6bd2c0eddf2bc9ffbce755c3c1e0290580b58765d9d0bdb2e5d9668df2f51183"
Jul 16 22:20:22.135740 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:22.135707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:23.053934 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:23.053851    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ecc4464-5753-4097-88d0-95540b665c09" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc" exitCode=1
Jul 16 22:20:23.054358 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:23.053941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" event={"ID":"8ecc4464-5753-4097-88d0-95540b665c09","Type":"ContainerDied","Data":"a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"}
Jul 16 22:20:23.054358 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:23.053999    2562 scope.go:117] "RemoveContainer" containerID="6bd2c0eddf2bc9ffbce755c3c1e0290580b58765d9d0bdb2e5d9668df2f51183"
Jul 16 22:20:23.054358 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:23.054248    2562 scope.go:117] "RemoveContainer" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"
Jul 16 22:20:23.054501 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:23.054478    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:24.060794 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:24.060767    2562 scope.go:117] "RemoveContainer" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"
Jul 16 22:20:24.061165 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:24.060942    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:25.552311 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:25.552274    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:25.553001 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:25.552686    2562 scope.go:117] "RemoveContainer" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"
Jul 16 22:20:25.553001 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:25.552877    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:30.552864 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:30.552813    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:30.553256 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:30.553201    2562 scope.go:117] "RemoveContainer" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"
Jul 16 22:20:30.553377 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:30.553358    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:41.745720 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:41.745691    2562 scope.go:117] "RemoveContainer" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"
Jul 16 22:20:43.144831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:43.144739    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ecc4464-5753-4097-88d0-95540b665c09" containerID="e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b" exitCode=1
Jul 16 22:20:43.144831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:43.144818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" event={"ID":"8ecc4464-5753-4097-88d0-95540b665c09","Type":"ContainerDied","Data":"e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b"}
Jul 16 22:20:43.145294 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:43.144879    2562 scope.go:117] "RemoveContainer" containerID="a52e04bb6d1d86c233178086b3e5fa5961aa3bb2878f1e59eb139282511bbfdc"
Jul 16 22:20:43.145294 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:43.145236    2562 scope.go:117] "RemoveContainer" containerID="e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b"
Jul 16 22:20:43.145445 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:43.145428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:45.552339 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:45.552295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:45.552775 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:45.552705    2562 scope.go:117] "RemoveContainer" containerID="e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b"
Jul 16 22:20:45.552904 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:45.552884    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:50.552737 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:50.552702    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:20:50.553124 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:50.553069    2562 scope.go:117] "RemoveContainer" containerID="e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b"
Jul 16 22:20:50.553242 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:20:50.553223    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:20:57.885321 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:20:57.885279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:01.132148 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:01.132107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:01.208475 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:01.208437    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:03.745071 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:03.745044    2562 scope.go:117] "RemoveContainer" containerID="e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b"
Jul 16 22:21:05.244494 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:05.244466    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ecc4464-5753-4097-88d0-95540b665c09" containerID="91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3" exitCode=1
Jul 16 22:21:05.244920 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:05.244562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" event={"ID":"8ecc4464-5753-4097-88d0-95540b665c09","Type":"ContainerDied","Data":"91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3"}
Jul 16 22:21:05.244920 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:05.244637    2562 scope.go:117] "RemoveContainer" containerID="e2a0825b3c908ecdf78b6429997b42324cd6d3e4f8d40be585d8c71bc169ea0b"
Jul 16 22:21:05.245024 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:05.244992    2562 scope.go:117] "RemoveContainer" containerID="91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3"
Jul 16 22:21:05.245211 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:05.245187    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:21:05.552188 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:05.552107    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:21:06.251405 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:06.251380    2562 scope.go:117] "RemoveContainer" containerID="91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3"
Jul 16 22:21:06.251861 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:06.251588    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:21:10.551848 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:10.551761    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:21:10.552215 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:10.552138    2562 scope.go:117] "RemoveContainer" containerID="91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3"
Jul 16 22:21:10.552341 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:10.552321    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:21:23.745948 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:23.745911    2562 scope.go:117] "RemoveContainer" containerID="91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3"
Jul 16 22:21:23.746454 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:23.746133    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch_odh-ai-gateway-infra(8ecc4464-5753-4097-88d0-95540b665c09)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" podUID="8ecc4464-5753-4097-88d0-95540b665c09"
Jul 16 22:21:24.824746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:24.824712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:24.918623 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:24.918591    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:31.053751 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.053718    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"]
Jul 16 22:21:31.084607 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.084577    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:31.185188 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.185166    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:21:31.291352 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.291319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvz22\" (UniqueName: \"kubernetes.io/projected/8ecc4464-5753-4097-88d0-95540b665c09-kube-api-access-pvz22\") pod \"8ecc4464-5753-4097-88d0-95540b665c09\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") "
Jul 16 22:21:31.291596 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.291452    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ecc4464-5753-4097-88d0-95540b665c09-maas-api-tls\") pod \"8ecc4464-5753-4097-88d0-95540b665c09\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") "
Jul 16 22:21:31.291596 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.291513    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4464-5753-4097-88d0-95540b665c09-openshift-service-ca\") pod \"8ecc4464-5753-4097-88d0-95540b665c09\" (UID: \"8ecc4464-5753-4097-88d0-95540b665c09\") "
Jul 16 22:21:31.291985 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.291940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecc4464-5753-4097-88d0-95540b665c09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ecc4464-5753-4097-88d0-95540b665c09" (UID: "8ecc4464-5753-4097-88d0-95540b665c09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:31.293949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.293919    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecc4464-5753-4097-88d0-95540b665c09-kube-api-access-pvz22" (OuterVolumeSpecName: "kube-api-access-pvz22") pod "8ecc4464-5753-4097-88d0-95540b665c09" (UID: "8ecc4464-5753-4097-88d0-95540b665c09"). InnerVolumeSpecName "kube-api-access-pvz22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:21:31.293949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.293925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecc4464-5753-4097-88d0-95540b665c09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ecc4464-5753-4097-88d0-95540b665c09" (UID: "8ecc4464-5753-4097-88d0-95540b665c09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:21:31.341309 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.341272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:31.365598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.365573    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"
Jul 16 22:21:31.365598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.365583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch" event={"ID":"8ecc4464-5753-4097-88d0-95540b665c09","Type":"ContainerDied","Data":"89357389c8c1b120a819789cc1690bd15a7b58307b52b50b605f42f498ad8f79"}
Jul 16 22:21:31.365788 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.365643    2562 scope.go:117] "RemoveContainer" containerID="91b370ab398272fb6b05915afd92c64f5e9bfd8cdf0343db85e528e2af17d2e3"
Jul 16 22:21:31.387833 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.387808    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"]
Jul 16 22:21:31.393122 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.393096    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ecc4464-5753-4097-88d0-95540b665c09-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:21:31.393219 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.393128    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecc4464-5753-4097-88d0-95540b665c09-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:21:31.393219 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.393147    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvz22\" (UniqueName: \"kubernetes.io/projected/8ecc4464-5753-4097-88d0-95540b665c09-kube-api-access-pvz22\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:21:31.393582 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.393540    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d06d9c2-6dd56d8c49-mbtch"]
Jul 16 22:21:31.680092 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.680035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:31.750453 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:31.750423    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecc4464-5753-4097-88d0-95540b665c09" path="/var/lib/kubelet/pods/8ecc4464-5753-4097-88d0-95540b665c09/volumes"
Jul 16 22:21:37.395425 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:37.395386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:37.469412 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:37.469380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:37.651183 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:37.651151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:37.750038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:37.750006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:38.841107 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.841062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:38.923155 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923122    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"]
Jul 16 22:21:38.923471 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923459    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923472    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923486    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923491    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923498    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923504    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd0f85-b8fa-483d-9bf0-e5a13b96745b" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923511    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923516    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923522 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923523    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923528    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923540    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923563    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923616    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923623    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.923823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.923629    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:21:38.927293 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.927268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:38.927721 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.927704    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:38.935057 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.935038    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-24cdbda3-serving-cert\""
Jul 16 22:21:38.935181 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.935043    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:21:38.935879 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.935864    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:21:38.935940 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.935899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:21:38.942202 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.942182    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"]
Jul 16 22:21:38.959366 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.959340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdcf\" (UniqueName: \"kubernetes.io/projected/371d106b-7931-42bc-ba06-2dce186dd89f-kube-api-access-dmdcf\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:38.959504 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.959378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:38.959504 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:38.959437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371d106b-7931-42bc-ba06-2dce186dd89f-openshift-service-ca\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.014923 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.014891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:39.060623 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.060592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371d106b-7931-42bc-ba06-2dce186dd89f-openshift-service-ca\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.060801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.060669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdcf\" (UniqueName: \"kubernetes.io/projected/371d106b-7931-42bc-ba06-2dce186dd89f-kube-api-access-dmdcf\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.060801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.060694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.060801 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:39.060788    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-serving-cert: secret "maas-api-e2e-ait-24cdbda3-serving-cert" not found
Jul 16 22:21:39.060930 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:39.060840    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls podName:371d106b-7931-42bc-ba06-2dce186dd89f nodeName:}" failed. No retries permitted until 2026-07-16 22:21:39.560824043 +0000 UTC m=+2082.342721869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls") pod "maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" (UID: "371d106b-7931-42bc-ba06-2dce186dd89f") : secret "maas-api-e2e-ait-24cdbda3-serving-cert" not found
Jul 16 22:21:39.061298 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.061279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371d106b-7931-42bc-ba06-2dce186dd89f-openshift-service-ca\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.073411 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.073387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdcf\" (UniqueName: \"kubernetes.io/projected/371d106b-7931-42bc-ba06-2dce186dd89f-kube-api-access-dmdcf\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.177188 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.177149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:39.565863 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.565758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.568581 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.568536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls\") pod \"maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.660053 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.660016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:39.759879 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.759841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:39.838244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.838144    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:39.977368 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:39.977341    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"]
Jul 16 22:21:39.981309 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:21:39.981270    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d106b_7931_42bc_ba06_2dce186dd89f.slice/crio-39ab7245cef283fb3007bf9f8f7f0e9e1403695eaf9e4a9415f8c3690b158f77 WatchSource:0}: Error finding container 39ab7245cef283fb3007bf9f8f7f0e9e1403695eaf9e4a9415f8c3690b158f77: Status 404 returned error can't find the container with id 39ab7245cef283fb3007bf9f8f7f0e9e1403695eaf9e4a9415f8c3690b158f77
Jul 16 22:21:40.411368 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:40.411327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerStarted","Data":"39ab7245cef283fb3007bf9f8f7f0e9e1403695eaf9e4a9415f8c3690b158f77"}
Jul 16 22:21:41.039595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:41.039560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:21:41.417566 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:41.417522    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="f30e5c285746598357d01092773833c398afc2dab5f173bdab3218375d06ef2c" exitCode=1
Jul 16 22:21:41.417738 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:41.417577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"f30e5c285746598357d01092773833c398afc2dab5f173bdab3218375d06ef2c"}
Jul 16 22:21:41.417845 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:41.417831    2562 scope.go:117] "RemoveContainer" containerID="f30e5c285746598357d01092773833c398afc2dab5f173bdab3218375d06ef2c"
Jul 16 22:21:42.424776 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:42.424740    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c" exitCode=1
Jul 16 22:21:42.425162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:42.424832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"}
Jul 16 22:21:42.425162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:42.424889    2562 scope.go:117] "RemoveContainer" containerID="f30e5c285746598357d01092773833c398afc2dab5f173bdab3218375d06ef2c"
Jul 16 22:21:42.425162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:42.425145    2562 scope.go:117] "RemoveContainer" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"
Jul 16 22:21:42.425357 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:42.425336    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:21:43.430750 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:43.430725    2562 scope.go:117] "RemoveContainer" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"
Jul 16 22:21:43.431107 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:43.430902    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:21:44.838753 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:44.838718    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:44.839152 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:44.839076    2562 scope.go:117] "RemoveContainer" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"
Jul 16 22:21:44.839285 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:44.839263    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:21:49.839374 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:49.839335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:21:49.839921 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:49.839902    2562 scope.go:117] "RemoveContainer" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"
Jul 16 22:21:49.840169 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:21:49.840144    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:21:54.117805 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:21:54.117772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:02.745482 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:02.745448    2562 scope.go:117] "RemoveContainer" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"
Jul 16 22:22:02.745985 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:02.745781    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:22:03.520852 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:03.520752    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerStarted","Data":"116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"}
Jul 16 22:22:03.521014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:03.520993    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:22:03.546515 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:03.546461    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podStartSLOduration=24.221024658 podStartE2EDuration="25.546445791s" totalImagesPullingTime="1.325421133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:03.544217 +0000 UTC m=+2106.326114850" watchObservedRunningTime="2026-07-16 22:22:03.546445791 +0000 UTC m=+2106.328343639"
Jul 16 22:22:04.526876 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:04.526844    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2" exitCode=1
Jul 16 22:22:04.527324 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:04.526940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"}
Jul 16 22:22:04.527324 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:04.527000    2562 scope.go:117] "RemoveContainer" containerID="4704db27f93a6620422457f932f56847fd595a39d2a31e3b3f3637263f081e3c"
Jul 16 22:22:04.527324 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:04.527169    2562 scope.go:117] "RemoveContainer" containerID="116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"
Jul 16 22:22:04.527499 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:04.527384    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:05.533313 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:05.533282    2562 scope.go:117] "RemoveContainer" containerID="116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"
Jul 16 22:22:05.533742 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:05.533449    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:09.838390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:09.838361    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:22:09.838818 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:09.838740    2562 scope.go:117] "RemoveContainer" containerID="116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"
Jul 16 22:22:09.838907 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:09.838888    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:23.745931 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:23.745893    2562 scope.go:117] "RemoveContainer" containerID="116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"
Jul 16 22:22:24.625538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:24.625508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerStarted","Data":"cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"}
Jul 16 22:22:24.625753 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:24.625735    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:22:25.631407 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:25.631375    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7" exitCode=1
Jul 16 22:22:25.631842 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:25.631469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"}
Jul 16 22:22:25.631842 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:25.631520    2562 scope.go:117] "RemoveContainer" containerID="116c54465e00c827925aa4d074674d9e152dd16dba6f0bfff3d7bbe9be3011a2"
Jul 16 22:22:25.631842 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:25.631687    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:22:25.631951 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:25.631894    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:26.637657 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:26.637630    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:22:26.638058 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:26.637802    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:29.838467 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:29.838424    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:22:29.838991 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:29.838920    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:22:29.839145 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:29.839121    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:41.745365 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:41.745337    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:22:41.745941 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:41.745519    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:22:53.745213 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:22:53.745116    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:22:53.745797 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:22:53.745325    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:23:04.745855 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:04.745821    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:23:04.746266 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:23:04.745992    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:23:18.745202 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:18.745172    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:23:19.869644 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:19.869611    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d" exitCode=1
Jul 16 22:23:19.870089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:19.869687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"}
Jul 16 22:23:19.870089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:19.869765    2562 scope.go:117] "RemoveContainer" containerID="cbc7b8fec57cb78e18298ae826d3d1b2b72162b320691d40b7a2463c6b5a2dd7"
Jul 16 22:23:19.870218 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:19.870167    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:23:19.870450 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:23:19.870426    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:23:21.403798 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:21.403770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:24.626470 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:24.626433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:23:24.626913 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:24.626862    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:23:24.627071 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:23:24.627049    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:23:29.839285 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:29.839251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:23:29.839769 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:29.839623    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:23:29.839811 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:23:29.839775    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:23:41.746103 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:41.746072    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:23:41.746492 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:23:41.746260    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:23:56.745773 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:23:56.745738    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:23:56.746171 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:23:56.745915    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:24:09.745895 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:09.745859    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:24:09.746506 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:24:09.746063    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:24:23.745422 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:23.745389    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:24:23.745852 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:24:23.745605    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:24:36.745934 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:36.745904    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:24:36.746335 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:24:36.746078    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:24:37.079650 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:37.079569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:24:47.747425 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:47.747396    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:24:49.271740 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:49.271709    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20" exitCode=1
Jul 16 22:24:49.272258 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:49.271781    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"}
Jul 16 22:24:49.272258 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:49.271860    2562 scope.go:117] "RemoveContainer" containerID="c59815f1745e5cbe1d9caeb74d02c10b4ccc57e48290b5605bd47912acf1373d"
Jul 16 22:24:49.272258 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:49.272193    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:24:49.272432 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:24:49.272397    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:24:49.626352 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:49.626266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:24:49.838324 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:49.838293    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:24:50.278107 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:50.278079    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:24:50.278493 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:24:50.278263    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:24:51.283578 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:24:51.283528    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:24:51.284063 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:24:51.283790    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:25:03.164624 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:03.164589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:25:03.350219 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:03.350182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:25:04.745680 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:04.745643    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:25:04.746113 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:25:04.745812    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:25:05.535687 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:05.535642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:25:05.636818 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:05.636785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:25:17.749082 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:17.749050    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:25:17.749579 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:25:17.749255    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:25:28.745667 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:28.745632    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:25:28.746122 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:25:28.745831    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:25:41.745348 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:41.745316    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:25:41.745867 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:25:41.745537    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:25:54.745090 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:25:54.744996    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:25:54.745628 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:25:54.745254    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:26:06.746016 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:06.745988    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:26:06.746469 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:26:06.746177    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:26:14.242301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:14.242259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:26:15.556722 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:15.556683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:26:21.745932 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:21.745900    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:26:21.746420 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:26:21.746075    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:26:24.202309 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:24.202278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:26:35.745530 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:35.745498    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:26:35.745960 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:26:35.745712    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:26:35.746631 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:35.746600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:26:45.218830 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:45.218791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:26:49.745846 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:26:49.745811    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:26:49.746232 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:26:49.746033    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:27:01.746027 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:01.745992    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:27:01.746516 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:27:01.746175    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:27:16.745275 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:16.745241    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:27:16.745735 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:27:16.745422    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:27:31.745939 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:31.745902    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:27:31.746329 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:31.746214    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:27:33.020333 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:33.020301    2562 generic.go:356] "Generic (PLEG): container finished" podID="371d106b-7931-42bc-ba06-2dce186dd89f" containerID="3a152e0380538d7d586444f1ffa001c4213d15f7aacde7c16db65dfaa4251586" exitCode=1
Jul 16 22:27:33.020806 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:33.020370    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"3a152e0380538d7d586444f1ffa001c4213d15f7aacde7c16db65dfaa4251586"}
Jul 16 22:27:33.020806 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:33.020431    2562 scope.go:117] "RemoveContainer" containerID="3782664f683864d342193237a8ab14fe49a2d0c05287c3446aa749b2cf197b20"
Jul 16 22:27:33.020943 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:33.020828    2562 scope.go:117] "RemoveContainer" containerID="3a152e0380538d7d586444f1ffa001c4213d15f7aacde7c16db65dfaa4251586"
Jul 16 22:27:33.021030 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:27:33.021013    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:27:34.626382 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:34.626348    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:27:34.626914 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:34.626837    2562 scope.go:117] "RemoveContainer" containerID="3a152e0380538d7d586444f1ffa001c4213d15f7aacde7c16db65dfaa4251586"
Jul 16 22:27:34.627071 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:27:34.627048    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:27:39.839213 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:39.839163    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:27:39.839640 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:39.839525    2562 scope.go:117] "RemoveContainer" containerID="3a152e0380538d7d586444f1ffa001c4213d15f7aacde7c16db65dfaa4251586"
Jul 16 22:27:39.839772 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:27:39.839749    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k_odh-ai-gateway-infra(371d106b-7931-42bc-ba06-2dce186dd89f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" podUID="371d106b-7931-42bc-ba06-2dce186dd89f"
Jul 16 22:27:44.187642 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:44.187583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:44.325913 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:44.325879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:50.730082 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.730044    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"]
Jul 16 22:27:50.761896 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.761862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:50.866637 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.866611    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:27:50.888059 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.888025    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:50.891149 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.891122    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmdcf\" (UniqueName: \"kubernetes.io/projected/371d106b-7931-42bc-ba06-2dce186dd89f-kube-api-access-dmdcf\") pod \"371d106b-7931-42bc-ba06-2dce186dd89f\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") "
Jul 16 22:27:50.891284 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.891210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371d106b-7931-42bc-ba06-2dce186dd89f-openshift-service-ca\") pod \"371d106b-7931-42bc-ba06-2dce186dd89f\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") "
Jul 16 22:27:50.891284 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.891250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls\") pod \"371d106b-7931-42bc-ba06-2dce186dd89f\" (UID: \"371d106b-7931-42bc-ba06-2dce186dd89f\") "
Jul 16 22:27:50.891608 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.891583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371d106b-7931-42bc-ba06-2dce186dd89f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "371d106b-7931-42bc-ba06-2dce186dd89f" (UID: "371d106b-7931-42bc-ba06-2dce186dd89f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:50.893439 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.893410    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d106b-7931-42bc-ba06-2dce186dd89f-kube-api-access-dmdcf" (OuterVolumeSpecName: "kube-api-access-dmdcf") pod "371d106b-7931-42bc-ba06-2dce186dd89f" (UID: "371d106b-7931-42bc-ba06-2dce186dd89f"). InnerVolumeSpecName "kube-api-access-dmdcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:50.893527 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.893458    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "371d106b-7931-42bc-ba06-2dce186dd89f" (UID: "371d106b-7931-42bc-ba06-2dce186dd89f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:50.992747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.992664    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371d106b-7931-42bc-ba06-2dce186dd89f-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:27:50.992747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.992700    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371d106b-7931-42bc-ba06-2dce186dd89f-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:27:50.992747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:50.992711    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdcf\" (UniqueName: \"kubernetes.io/projected/371d106b-7931-42bc-ba06-2dce186dd89f-kube-api-access-dmdcf\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:27:51.101788 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:51.101762    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"
Jul 16 22:27:51.101949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:51.101800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k" event={"ID":"371d106b-7931-42bc-ba06-2dce186dd89f","Type":"ContainerDied","Data":"39ab7245cef283fb3007bf9f8f7f0e9e1403695eaf9e4a9415f8c3690b158f77"}
Jul 16 22:27:51.101949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:51.101861    2562 scope.go:117] "RemoveContainer" containerID="3a152e0380538d7d586444f1ffa001c4213d15f7aacde7c16db65dfaa4251586"
Jul 16 22:27:51.124616 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:51.124590    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"]
Jul 16 22:27:51.128162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:51.128138    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24cdbda3-9d96f5f65-7gj8k"]
Jul 16 22:27:51.752996 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:51.752955    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" path="/var/lib/kubelet/pods/371d106b-7931-42bc-ba06-2dce186dd89f/volumes"
Jul 16 22:27:56.852040 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:56.852010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:56.950685 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:56.950657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:57.154850 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:57.154812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:58.378913 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.378820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:58.516485 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.516450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:58.581864 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.581829    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"]
Jul 16 22:27:58.582159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582148    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582161    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582171    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582176    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582182    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582188    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582200    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582204    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582210    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582215    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582274    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582281    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582287    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582292    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582300    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecc4464-5753-4097-88d0-95540b665c09" containerName="maas-api"
Jul 16 22:27:58.582415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.582308    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:27:58.586678 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.586661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.589485 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.589465    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a6237863-serving-cert\""
Jul 16 22:27:58.590250 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.590235    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:27:58.590587 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.590568    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:27:58.599788 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.599766    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:27:58.617766 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.617734    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"]
Jul 16 22:27:58.658412 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.658384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzc4c\" (UniqueName: \"kubernetes.io/projected/57e2f853-b214-4796-8579-d936a9fbf8b5-kube-api-access-gzc4c\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.658598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.658436    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2f853-b214-4796-8579-d936a9fbf8b5-openshift-service-ca\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.658598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.658527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2f853-b214-4796-8579-d936a9fbf8b5-maas-api-tls\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.676694 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.676645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:58.759153 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.759123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2f853-b214-4796-8579-d936a9fbf8b5-openshift-service-ca\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.759327 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.759167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2f853-b214-4796-8579-d936a9fbf8b5-maas-api-tls\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.759327 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.759215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzc4c\" (UniqueName: \"kubernetes.io/projected/57e2f853-b214-4796-8579-d936a9fbf8b5-kube-api-access-gzc4c\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.759953 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.759922    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2f853-b214-4796-8579-d936a9fbf8b5-openshift-service-ca\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.761769 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.761750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2f853-b214-4796-8579-d936a9fbf8b5-maas-api-tls\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.768008 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.767986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzc4c\" (UniqueName: \"kubernetes.io/projected/57e2f853-b214-4796-8579-d936a9fbf8b5-kube-api-access-gzc4c\") pod \"maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:58.869816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.869782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:58.898207 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:58.898171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:27:59.033879 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:59.033839    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"]
Jul 16 22:27:59.038202 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:27:59.037212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e2f853_b214_4796_8579_d936a9fbf8b5.slice/crio-0320d37953a2f6db842b9c20d27cebe80e28cf12097df35eb7188432b050ee8e WatchSource:0}: Error finding container 0320d37953a2f6db842b9c20d27cebe80e28cf12097df35eb7188432b050ee8e: Status 404 returned error can't find the container with id 0320d37953a2f6db842b9c20d27cebe80e28cf12097df35eb7188432b050ee8e
Jul 16 22:27:59.146764 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:59.146666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerStarted","Data":"0320d37953a2f6db842b9c20d27cebe80e28cf12097df35eb7188432b050ee8e"}
Jul 16 22:27:59.314369 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:59.314280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:59.416832 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:27:59.416789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:28:00.153645 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:00.153611    2562 generic.go:356] "Generic (PLEG): container finished" podID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerID="0faaffcd91fbb8cf979f07669b297f1a1f295fe8cbb1b0dd612965a380bd30dc" exitCode=1
Jul 16 22:28:00.153645 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:00.153655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"0faaffcd91fbb8cf979f07669b297f1a1f295fe8cbb1b0dd612965a380bd30dc"}
Jul 16 22:28:00.153954 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:00.153939    2562 scope.go:117] "RemoveContainer" containerID="0faaffcd91fbb8cf979f07669b297f1a1f295fe8cbb1b0dd612965a380bd30dc"
Jul 16 22:28:00.598440 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:00.598406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:28:00.699532 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:00.699495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:28:01.161541 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:01.161506    2562 generic.go:356] "Generic (PLEG): container finished" podID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a" exitCode=1
Jul 16 22:28:01.161734 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:01.161609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"}
Jul 16 22:28:01.161734 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:01.161724    2562 scope.go:117] "RemoveContainer" containerID="0faaffcd91fbb8cf979f07669b297f1a1f295fe8cbb1b0dd612965a380bd30dc"
Jul 16 22:28:01.162000 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:01.161983    2562 scope.go:117] "RemoveContainer" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"
Jul 16 22:28:01.162223 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:01.162204    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:02.167980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:02.167952    2562 scope.go:117] "RemoveContainer" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"
Jul 16 22:28:02.168362 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:02.168135    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:03.899183 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:03.899146    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:28:03.899683 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:03.899536    2562 scope.go:117] "RemoveContainer" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"
Jul 16 22:28:03.899762 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:03.899742    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:04.310535 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:04.310497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:28:08.899089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:08.899038    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:28:08.899496 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:08.899457    2562 scope.go:117] "RemoveContainer" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"
Jul 16 22:28:08.899691 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:08.899668    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:21.746026 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:21.745991    2562 scope.go:117] "RemoveContainer" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"
Jul 16 22:28:22.258778 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:22.258744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerStarted","Data":"cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"}
Jul 16 22:28:22.258984 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:22.258967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:28:22.281397 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:22.281353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podStartSLOduration=22.845859838 podStartE2EDuration="24.281338895s" totalImagesPullingTime="1.435479057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:22.280420918 +0000 UTC m=+2485.062318766" watchObservedRunningTime="2026-07-16 22:28:22.281338895 +0000 UTC m=+2485.063236742"
Jul 16 22:28:23.265403 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:23.265366    2562 generic.go:356] "Generic (PLEG): container finished" podID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerID="cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a" exitCode=1
Jul 16 22:28:23.265927 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:23.265459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"}
Jul 16 22:28:23.265927 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:23.265532    2562 scope.go:117] "RemoveContainer" containerID="85edb9ddad74e31bb07e43af52b446ad288ac58e763de961ba354d3d1a0fc33a"
Jul 16 22:28:23.265927 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:23.265796    2562 scope.go:117] "RemoveContainer" containerID="cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"
Jul 16 22:28:23.266070 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:23.265997    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:24.271669 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:24.271639    2562 scope.go:117] "RemoveContainer" containerID="cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"
Jul 16 22:28:24.272085 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:24.271828    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:28.898621 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:28.898584    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:28:28.899119 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:28.899079    2562 scope.go:117] "RemoveContainer" containerID="cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"
Jul 16 22:28:28.899324 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:28.899299    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:42.745311 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:42.745277    2562 scope.go:117] "RemoveContainer" containerID="cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"
Jul 16 22:28:43.354651 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:43.354567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerStarted","Data":"e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"}
Jul 16 22:28:43.354817 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:43.354804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:28:44.361386 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:44.361349    2562 generic.go:356] "Generic (PLEG): container finished" podID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9" exitCode=1
Jul 16 22:28:44.361854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:44.361458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"}
Jul 16 22:28:44.361854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:44.361542    2562 scope.go:117] "RemoveContainer" containerID="cfea8f3d26928f891c9f28c47a34f67fede30883d3e9aa305b9b50e52895b69a"
Jul 16 22:28:44.361854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:44.361739    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:28:44.361989 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:44.361951    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:45.367718 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:45.367694    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:28:45.368120 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:45.367867    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:28:48.899280 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:48.899190    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:28:48.899694 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:28:48.899614    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:28:48.899800 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:28:48.899780    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:29:02.745523 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:02.745483    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:29:02.745973 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:29:02.745704    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:29:16.745896 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:16.745858    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:29:16.746362 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:29:16.746086    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:29:27.746984 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:27.746952    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:29:28.563538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:28.563500    2562 generic.go:356] "Generic (PLEG): container finished" podID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a" exitCode=1
Jul 16 22:29:28.563752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:28.563584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"}
Jul 16 22:29:28.563752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:28.563654    2562 scope.go:117] "RemoveContainer" containerID="e39fc3d2797e36469f861978e67ddb11a8e3051616aa88a008ce5ab3aecbede9"
Jul 16 22:29:28.564032 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:28.564009    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:29:28.564244 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:29:28.564222    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:29:28.899129 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:28.899094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:29:29.570375 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:29.570344    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:29:29.570579 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:29:29.570527    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:29:33.355443 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:33.355406    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:29:33.355886 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:33.355846    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:29:33.356053 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:29:33.356031    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:29:46.745779 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:29:46.745745    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:29:46.746199 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:29:46.745918    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:00.154846 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.154806    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"]
Jul 16 22:30:00.155427 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.155312    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:30:00.155427 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.155331    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:30:00.155427 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.155343    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:30:00.155427 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.155352    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:30:00.155682 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.155449    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:30:00.155682 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.155460    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="371d106b-7931-42bc-ba06-2dce186dd89f" containerName="maas-api"
Jul 16 22:30:00.158563 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.158526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"
Jul 16 22:30:00.166860 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.166838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:30:00.168232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.168216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"
Jul 16 22:30:00.178581 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.178534    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"]
Jul 16 22:30:00.328228 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.328168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"]
Jul 16 22:30:00.330465 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:30:00.330441    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab60dd2_1cc9_4d1a_9f9b_4ed70cb99fe7.slice/crio-dc188cb54088684228625a9c5be4d1430bc71361ba4d8ac0bbbd44ba29cec0f1 WatchSource:0}: Error finding container dc188cb54088684228625a9c5be4d1430bc71361ba4d8ac0bbbd44ba29cec0f1: Status 404 returned error can't find the container with id dc188cb54088684228625a9c5be4d1430bc71361ba4d8ac0bbbd44ba29cec0f1
Jul 16 22:30:00.710633 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.710595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8" event={"ID":"6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7","Type":"ContainerStarted","Data":"d4b76c03b157ece6029b00d894f37648fef5d025f4a5711a7b3b4f39aecb4319"}
Jul 16 22:30:00.710772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.710659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8" event={"ID":"6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7","Type":"ContainerStarted","Data":"dc188cb54088684228625a9c5be4d1430bc71361ba4d8ac0bbbd44ba29cec0f1"}
Jul 16 22:30:00.742342 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:00.742289    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8" podStartSLOduration=0.74227284 podStartE2EDuration="742.27284ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:00.741579732 +0000 UTC m=+2583.523477584" watchObservedRunningTime="2026-07-16 22:30:00.74227284 +0000 UTC m=+2583.524170692"
Jul 16 22:30:01.716345 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:01.716309    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7" containerID="d4b76c03b157ece6029b00d894f37648fef5d025f4a5711a7b3b4f39aecb4319" exitCode=0
Jul 16 22:30:01.716752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:01.716396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8" event={"ID":"6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7","Type":"ContainerDied","Data":"d4b76c03b157ece6029b00d894f37648fef5d025f4a5711a7b3b4f39aecb4319"}
Jul 16 22:30:01.746111 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:01.746080    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:30:01.746324 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:01.746300    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:02.852664 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:02.852635    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"
Jul 16 22:30:03.726770 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:03.726743    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8"
Jul 16 22:30:03.726987 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:03.726785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-4t7h8" event={"ID":"6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7","Type":"ContainerDied","Data":"dc188cb54088684228625a9c5be4d1430bc71361ba4d8ac0bbbd44ba29cec0f1"}
Jul 16 22:30:03.726987 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:03.726832    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc188cb54088684228625a9c5be4d1430bc71361ba4d8ac0bbbd44ba29cec0f1"
Jul 16 22:30:04.196193 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:04.196157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:04.283853 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:04.283819    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:12.745499 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:12.745460    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:30:12.747851 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:12.745753    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:13.722076 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:13.722037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:13.877226 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:13.877197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:25.745054 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:25.745026    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:30:25.745444 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:25.745219    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:37.747007 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:37.746969    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:30:37.747481 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:37.747221    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:50.746091 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:50.746064    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:30:51.935400 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:51.935370    2562 generic.go:356] "Generic (PLEG): container finished" podID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerID="1f95ca7013cc395656e9c49115e82d02ef1bdf131bc3c0d9c061ca3a15a84856" exitCode=1
Jul 16 22:30:51.935801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:51.935430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"1f95ca7013cc395656e9c49115e82d02ef1bdf131bc3c0d9c061ca3a15a84856"}
Jul 16 22:30:51.935801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:51.935485    2562 scope.go:117] "RemoveContainer" containerID="217abfff6c59fd7e596dab2657fe20429206f58e65a3e74ad17369d6b97bc16a"
Jul 16 22:30:51.935899 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:51.935817    2562 scope.go:117] "RemoveContainer" containerID="1f95ca7013cc395656e9c49115e82d02ef1bdf131bc3c0d9c061ca3a15a84856"
Jul 16 22:30:51.936031 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:51.936010    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:53.354904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:53.354867    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:30:53.355357 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:53.355240    2562 scope.go:117] "RemoveContainer" containerID="1f95ca7013cc395656e9c49115e82d02ef1bdf131bc3c0d9c061ca3a15a84856"
Jul 16 22:30:53.355422 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:53.355400    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:30:58.898469 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:58.898433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:30:58.898897 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:30:58.898848    2562 scope.go:117] "RemoveContainer" containerID="1f95ca7013cc395656e9c49115e82d02ef1bdf131bc3c0d9c061ca3a15a84856"
Jul 16 22:30:58.899041 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:30:58.899022    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq_odh-ai-gateway-infra(57e2f853-b214-4796-8579-d936a9fbf8b5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5"
Jul 16 22:31:02.824885 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:02.824837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:02.925444 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:02.925404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:09.102746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.102714    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"]
Jul 16 22:31:09.133009 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.132979    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:09.237054 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.237026    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:31:09.420336 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.420304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2f853-b214-4796-8579-d936a9fbf8b5-maas-api-tls\") pod \"57e2f853-b214-4796-8579-d936a9fbf8b5\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") "
Jul 16 22:31:09.420530 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.420354    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzc4c\" (UniqueName: \"kubernetes.io/projected/57e2f853-b214-4796-8579-d936a9fbf8b5-kube-api-access-gzc4c\") pod \"57e2f853-b214-4796-8579-d936a9fbf8b5\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") "
Jul 16 22:31:09.420530 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.420462    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2f853-b214-4796-8579-d936a9fbf8b5-openshift-service-ca\") pod \"57e2f853-b214-4796-8579-d936a9fbf8b5\" (UID: \"57e2f853-b214-4796-8579-d936a9fbf8b5\") "
Jul 16 22:31:09.420838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.420811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e2f853-b214-4796-8579-d936a9fbf8b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57e2f853-b214-4796-8579-d936a9fbf8b5" (UID: "57e2f853-b214-4796-8579-d936a9fbf8b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:09.422514 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.422493    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e2f853-b214-4796-8579-d936a9fbf8b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57e2f853-b214-4796-8579-d936a9fbf8b5" (UID: "57e2f853-b214-4796-8579-d936a9fbf8b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:31:09.422657 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.422636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e2f853-b214-4796-8579-d936a9fbf8b5-kube-api-access-gzc4c" (OuterVolumeSpecName: "kube-api-access-gzc4c") pod "57e2f853-b214-4796-8579-d936a9fbf8b5" (UID: "57e2f853-b214-4796-8579-d936a9fbf8b5"). InnerVolumeSpecName "kube-api-access-gzc4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:09.479834 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.479804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:09.522183 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.522148    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2f853-b214-4796-8579-d936a9fbf8b5-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:31:09.522183 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.522182    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2f853-b214-4796-8579-d936a9fbf8b5-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:31:09.522358 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:09.522193    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzc4c\" (UniqueName: \"kubernetes.io/projected/57e2f853-b214-4796-8579-d936a9fbf8b5-kube-api-access-gzc4c\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:31:10.016319 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:10.016241    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"
Jul 16 22:31:10.016319 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:10.016277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq" event={"ID":"57e2f853-b214-4796-8579-d936a9fbf8b5","Type":"ContainerDied","Data":"0320d37953a2f6db842b9c20d27cebe80e28cf12097df35eb7188432b050ee8e"}
Jul 16 22:31:10.016516 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:10.016333    2562 scope.go:117] "RemoveContainer" containerID="1f95ca7013cc395656e9c49115e82d02ef1bdf131bc3c0d9c061ca3a15a84856"
Jul 16 22:31:10.032014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:10.031991    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"]
Jul 16 22:31:10.037928 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:10.037909    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a6237863-75774bbbc5-r2hfq"]
Jul 16 22:31:11.752698 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:11.752665    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" path="/var/lib/kubelet/pods/57e2f853-b214-4796-8579-d936a9fbf8b5/volumes"
Jul 16 22:31:15.416776 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:15.416745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:15.513845 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:15.513811    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:17.108678 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:17.108645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:17.807095 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:17.807053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:19.396239 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:19.396205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:24.092021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:24.091982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:24.204077 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:24.204035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:24.297508 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:24.297477    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:24.396106 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:24.396020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:24.800420 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:24.800380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:50.885085 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:50.885006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:58.652769 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:58.652728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:58.898678 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:58.898647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:31:59.085040 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:31:59.084954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:14.969463 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:14.969427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:22.236777 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:22.236746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:22.387734 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:22.387698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:32.945463 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:32.945423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:33.737832 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:33.737795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:39.974801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:39.974763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:40.088564 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:40.088520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:40.180766 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:40.180730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:40.323694 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:40.323605    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:41.062794 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:41.062727    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:41.162260 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:41.162230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:41.263370 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:41.263339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:42.931908 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:42.931876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:49.224570 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:49.224518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:49.324251 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:49.324215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:49.916757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:49.916719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:56.512101 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:56.512060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:56.605649 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:56.605618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:56.736319 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:56.736283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:57.090351 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:57.090308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:57.271793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:32:57.271763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:00.279446 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.279410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:00.404248 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.404216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:00.431024 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.430993    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"]
Jul 16 22:33:00.431535 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431515    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431540    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431572    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7" containerName="cleanup"
Jul 16 22:33:00.431603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431581    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7" containerName="cleanup"
Jul 16 22:33:00.431603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431600    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431730 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431609    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431730 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431618    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431730 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431625    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431730 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431634    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431730 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431642    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431878 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431748    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431878 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431758    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431878 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431768    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431878 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431778    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:00.431878 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.431789    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7" containerName="cleanup"
Jul 16 22:33:00.434430 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.434413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.439352 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.439334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50ab3915-serving-cert\""
Jul 16 22:33:00.439486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.439459    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:33:00.439575 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.439487    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:00.440471 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.440454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:33:00.450477 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.450454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"]
Jul 16 22:33:00.460674 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.460647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.460791 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.460737    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8b2\" (UniqueName: \"kubernetes.io/projected/a70eaa5d-0c2e-47eb-826c-acc878376779-kube-api-access-7n8b2\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.460846 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.460819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaa5d-0c2e-47eb-826c-acc878376779-openshift-service-ca\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.517380 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.517338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:00.561963 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.561868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8b2\" (UniqueName: \"kubernetes.io/projected/a70eaa5d-0c2e-47eb-826c-acc878376779-kube-api-access-7n8b2\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.561963 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.561911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaa5d-0c2e-47eb-826c-acc878376779-openshift-service-ca\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.562192 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.561972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.562192 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:33:00.562132    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-serving-cert: secret "maas-api-e2e-mt-50ab3915-serving-cert" not found
Jul 16 22:33:00.562305 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:33:00.562218    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls podName:a70eaa5d-0c2e-47eb-826c-acc878376779 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:01.062194388 +0000 UTC m=+2763.844092214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls") pod "maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" (UID: "a70eaa5d-0c2e-47eb-826c-acc878376779") : secret "maas-api-e2e-mt-50ab3915-serving-cert" not found
Jul 16 22:33:00.562624 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.562605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaa5d-0c2e-47eb-826c-acc878376779-openshift-service-ca\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.571291 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.571268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8b2\" (UniqueName: \"kubernetes.io/projected/a70eaa5d-0c2e-47eb-826c-acc878376779-kube-api-access-7n8b2\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:00.813307 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:00.813198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:01.021991 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.021958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:01.065525 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.065435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:01.068016 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.067986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls\") pod \"maas-api-e2e-mt-50ab3915-b587b4849-rqcjs\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:01.125769 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.125733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:01.345107 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.345012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:01.494835 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.494804    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"]
Jul 16 22:33:01.497923 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:33:01.497894    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70eaa5d_0c2e_47eb_826c_acc878376779.slice/crio-57014c879347138f7da0d0606a8531ad9bc4deda95ffa729e88593b3d0d2dd46 WatchSource:0}: Error finding container 57014c879347138f7da0d0606a8531ad9bc4deda95ffa729e88593b3d0d2dd46: Status 404 returned error can't find the container with id 57014c879347138f7da0d0606a8531ad9bc4deda95ffa729e88593b3d0d2dd46
Jul 16 22:33:01.499217 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.499200    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:33:01.509983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:01.509957    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" event={"ID":"a70eaa5d-0c2e-47eb-826c-acc878376779","Type":"ContainerStarted","Data":"57014c879347138f7da0d0606a8531ad9bc4deda95ffa729e88593b3d0d2dd46"}
Jul 16 22:33:02.516348 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:02.516319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" event={"ID":"a70eaa5d-0c2e-47eb-826c-acc878376779","Type":"ContainerStarted","Data":"0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f"}
Jul 16 22:33:02.516736 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:02.516456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:02.557743 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:02.557690    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" podStartSLOduration=2.082419918 podStartE2EDuration="2.557674156s" totalImagesPullingTime="475.254238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:02.556154533 +0000 UTC m=+2765.338052382" watchObservedRunningTime="2026-07-16 22:33:02.557674156 +0000 UTC m=+2765.339572004"
Jul 16 22:33:04.703078 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:04.703041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:05.373475 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:05.373444    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:05.459122 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:05.459087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:05.782897 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:05.782867    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:08.526531 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:08.526499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:12.154865 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:12.154824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:12.251681 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:12.251641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:23.048288 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.048184    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"]
Jul 16 22:33:23.048794 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.048426    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" podUID="a70eaa5d-0c2e-47eb-826c-acc878376779" containerName="maas-api" containerID="cri-o://0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f" gracePeriod=30
Jul 16 22:33:23.092513 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.092487    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:23.289179 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.289157    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:23.363982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.363894    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaa5d-0c2e-47eb-826c-acc878376779-openshift-service-ca\") pod \"a70eaa5d-0c2e-47eb-826c-acc878376779\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") "
Jul 16 22:33:23.363982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.363963    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n8b2\" (UniqueName: \"kubernetes.io/projected/a70eaa5d-0c2e-47eb-826c-acc878376779-kube-api-access-7n8b2\") pod \"a70eaa5d-0c2e-47eb-826c-acc878376779\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") "
Jul 16 22:33:23.364158 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.364054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls\") pod \"a70eaa5d-0c2e-47eb-826c-acc878376779\" (UID: \"a70eaa5d-0c2e-47eb-826c-acc878376779\") "
Jul 16 22:33:23.364270 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.364248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70eaa5d-0c2e-47eb-826c-acc878376779-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a70eaa5d-0c2e-47eb-826c-acc878376779" (UID: "a70eaa5d-0c2e-47eb-826c-acc878376779"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:23.364337 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.364325    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaa5d-0c2e-47eb-826c-acc878376779-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:33:23.366292 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.366260    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a70eaa5d-0c2e-47eb-826c-acc878376779" (UID: "a70eaa5d-0c2e-47eb-826c-acc878376779"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:33:23.366407 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.366336    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70eaa5d-0c2e-47eb-826c-acc878376779-kube-api-access-7n8b2" (OuterVolumeSpecName: "kube-api-access-7n8b2") pod "a70eaa5d-0c2e-47eb-826c-acc878376779" (UID: "a70eaa5d-0c2e-47eb-826c-acc878376779"). InnerVolumeSpecName "kube-api-access-7n8b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:33:23.465687 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.465651    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n8b2\" (UniqueName: \"kubernetes.io/projected/a70eaa5d-0c2e-47eb-826c-acc878376779-kube-api-access-7n8b2\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:33:23.465687 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.465680    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70eaa5d-0c2e-47eb-826c-acc878376779-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:33:23.619714 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.619623    2562 generic.go:356] "Generic (PLEG): container finished" podID="a70eaa5d-0c2e-47eb-826c-acc878376779" containerID="0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f" exitCode=0
Jul 16 22:33:23.619714 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.619685    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"
Jul 16 22:33:23.619951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.619711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" event={"ID":"a70eaa5d-0c2e-47eb-826c-acc878376779","Type":"ContainerDied","Data":"0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f"}
Jul 16 22:33:23.619951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.619795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs" event={"ID":"a70eaa5d-0c2e-47eb-826c-acc878376779","Type":"ContainerDied","Data":"57014c879347138f7da0d0606a8531ad9bc4deda95ffa729e88593b3d0d2dd46"}
Jul 16 22:33:23.619951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.619824    2562 scope.go:117] "RemoveContainer" containerID="0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f"
Jul 16 22:33:23.629323 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.629291    2562 scope.go:117] "RemoveContainer" containerID="0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f"
Jul 16 22:33:23.629724 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:33:23.629695    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f\": container with ID starting with 0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f not found: ID does not exist" containerID="0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f"
Jul 16 22:33:23.629796 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.629741    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f"} err="failed to get container status \"0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f\": rpc error: code = NotFound desc = could not find container \"0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f\": container with ID starting with 0e501b5424a8f769bbc62a7fefc66958d381b242cedbfd53bfcf4d6541d94e5f not found: ID does not exist"
Jul 16 22:33:23.642047 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.642016    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"]
Jul 16 22:33:23.649452 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.649428    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50ab3915-b587b4849-rqcjs"]
Jul 16 22:33:23.751158 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:23.751118    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70eaa5d-0c2e-47eb-826c-acc878376779" path="/var/lib/kubelet/pods/a70eaa5d-0c2e-47eb-826c-acc878376779/volumes"
Jul 16 22:33:27.710505 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:27.710470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:33.825120 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:33.825086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:33.889339 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:33.889302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:35.439697 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.439669    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"]
Jul 16 22:33:35.440093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.439994    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:35.440093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440005    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:35.440093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440014    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:35.440093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440019    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:35.440093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440044    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70eaa5d-0c2e-47eb-826c-acc878376779" containerName="maas-api"
Jul 16 22:33:35.440093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440052    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70eaa5d-0c2e-47eb-826c-acc878376779" containerName="maas-api"
Jul 16 22:33:35.440301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440103    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:35.440301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440112    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2f853-b214-4796-8579-d936a9fbf8b5" containerName="maas-api"
Jul 16 22:33:35.440301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.440124    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70eaa5d-0c2e-47eb-826c-acc878376779" containerName="maas-api"
Jul 16 22:33:35.443084 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.443063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:35.450510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.450483    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:33:35.450936 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.450664    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:35.455278 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.455256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"]
Jul 16 22:33:35.567272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.567236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083ea756-54b0-4e5f-8018-55854a1764ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd\" (UID: \"083ea756-54b0-4e5f-8018-55854a1764ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:35.668038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.667988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083ea756-54b0-4e5f-8018-55854a1764ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd\" (UID: \"083ea756-54b0-4e5f-8018-55854a1764ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:35.668648 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.668629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083ea756-54b0-4e5f-8018-55854a1764ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd\" (UID: \"083ea756-54b0-4e5f-8018-55854a1764ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:35.760609 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.760517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:35.895760 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:35.895734    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"]
Jul 16 22:33:35.897038 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:33:35.897008    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083ea756_54b0_4e5f_8018_55854a1764ff.slice/crio-9f145c9a5b6038f5aa52bebfac0a727cdd6c9a61fa5e93f3d951751b96a26140 WatchSource:0}: Error finding container 9f145c9a5b6038f5aa52bebfac0a727cdd6c9a61fa5e93f3d951751b96a26140: Status 404 returned error can't find the container with id 9f145c9a5b6038f5aa52bebfac0a727cdd6c9a61fa5e93f3d951751b96a26140
Jul 16 22:33:36.680988 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:36.680953    2562 generic.go:356] "Generic (PLEG): container finished" podID="083ea756-54b0-4e5f-8018-55854a1764ff" containerID="9df32afad8e50fe60bbcd56ea12a69c2a19815964cb4d6e9f827becd3f9a67a0" exitCode=6
Jul 16 22:33:36.680988 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:36.680990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" event={"ID":"083ea756-54b0-4e5f-8018-55854a1764ff","Type":"ContainerDied","Data":"9df32afad8e50fe60bbcd56ea12a69c2a19815964cb4d6e9f827becd3f9a67a0"}
Jul 16 22:33:36.681394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:36.681033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" event={"ID":"083ea756-54b0-4e5f-8018-55854a1764ff","Type":"ContainerStarted","Data":"9f145c9a5b6038f5aa52bebfac0a727cdd6c9a61fa5e93f3d951751b96a26140"}
Jul 16 22:33:36.681394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:36.681231    2562 scope.go:117] "RemoveContainer" containerID="9df32afad8e50fe60bbcd56ea12a69c2a19815964cb4d6e9f827becd3f9a67a0"
Jul 16 22:33:37.688692 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:37.688654    2562 generic.go:356] "Generic (PLEG): container finished" podID="083ea756-54b0-4e5f-8018-55854a1764ff" containerID="030cacf240f5055c86a0f50f1c89faf6f201f94434a377553e0b3fe845de66f2" exitCode=6
Jul 16 22:33:37.689151 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:37.688746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" event={"ID":"083ea756-54b0-4e5f-8018-55854a1764ff","Type":"ContainerDied","Data":"030cacf240f5055c86a0f50f1c89faf6f201f94434a377553e0b3fe845de66f2"}
Jul 16 22:33:37.689151 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:37.688813    2562 scope.go:117] "RemoveContainer" containerID="9df32afad8e50fe60bbcd56ea12a69c2a19815964cb4d6e9f827becd3f9a67a0"
Jul 16 22:33:37.689151 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:37.689061    2562 scope.go:117] "RemoveContainer" containerID="030cacf240f5055c86a0f50f1c89faf6f201f94434a377553e0b3fe845de66f2"
Jul 16 22:33:37.689340 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:33:37.689291    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd_odh-ai-gateway-infra(083ea756-54b0-4e5f-8018-55854a1764ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" podUID="083ea756-54b0-4e5f-8018-55854a1764ff"
Jul 16 22:33:38.559421 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:38.559386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:38.641089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:38.641054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:38.695060 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:38.695033    2562 scope.go:117] "RemoveContainer" containerID="030cacf240f5055c86a0f50f1c89faf6f201f94434a377553e0b3fe845de66f2"
Jul 16 22:33:38.695512 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:33:38.695279    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd_odh-ai-gateway-infra(083ea756-54b0-4e5f-8018-55854a1764ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" podUID="083ea756-54b0-4e5f-8018-55854a1764ff"
Jul 16 22:33:38.767083 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:38.767055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:38.861186 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:38.861087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:38.964347 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:38.964315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:45.874419 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:45.874383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:45.977078 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:45.977046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:46.082219 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:46.082183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:46.238418 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:46.238377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:49.745109 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:49.745079    2562 scope.go:117] "RemoveContainer" containerID="030cacf240f5055c86a0f50f1c89faf6f201f94434a377553e0b3fe845de66f2"
Jul 16 22:33:50.749355 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:50.749324    2562 generic.go:356] "Generic (PLEG): container finished" podID="083ea756-54b0-4e5f-8018-55854a1764ff" containerID="cdcca673d6492fd25653ecd4c0fe099effd3e5098fa7576f18b47c9a69656593" exitCode=6
Jul 16 22:33:50.749355 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:50.749360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" event={"ID":"083ea756-54b0-4e5f-8018-55854a1764ff","Type":"ContainerDied","Data":"cdcca673d6492fd25653ecd4c0fe099effd3e5098fa7576f18b47c9a69656593"}
Jul 16 22:33:50.749868 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:50.749408    2562 scope.go:117] "RemoveContainer" containerID="030cacf240f5055c86a0f50f1c89faf6f201f94434a377553e0b3fe845de66f2"
Jul 16 22:33:50.749868 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:50.749736    2562 scope.go:117] "RemoveContainer" containerID="cdcca673d6492fd25653ecd4c0fe099effd3e5098fa7576f18b47c9a69656593"
Jul 16 22:33:50.749981 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:33:50.749954    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd_odh-ai-gateway-infra(083ea756-54b0-4e5f-8018-55854a1764ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" podUID="083ea756-54b0-4e5f-8018-55854a1764ff"
Jul 16 22:33:51.775373 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:51.775337    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"]
Jul 16 22:33:51.904908 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:51.904885    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:52.012458 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.012425    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083ea756-54b0-4e5f-8018-55854a1764ff-maas-api-service-ca\") pod \"083ea756-54b0-4e5f-8018-55854a1764ff\" (UID: \"083ea756-54b0-4e5f-8018-55854a1764ff\") "
Jul 16 22:33:52.012800 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.012775    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083ea756-54b0-4e5f-8018-55854a1764ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "083ea756-54b0-4e5f-8018-55854a1764ff" (UID: "083ea756-54b0-4e5f-8018-55854a1764ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:52.113869 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.113780    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/083ea756-54b0-4e5f-8018-55854a1764ff-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:33:52.189053 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.189023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:52.289659 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.289621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:52.760570 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.760516    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"
Jul 16 22:33:52.760775 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.760512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd" event={"ID":"083ea756-54b0-4e5f-8018-55854a1764ff","Type":"ContainerDied","Data":"9f145c9a5b6038f5aa52bebfac0a727cdd6c9a61fa5e93f3d951751b96a26140"}
Jul 16 22:33:52.760775 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.760656    2562 scope.go:117] "RemoveContainer" containerID="cdcca673d6492fd25653ecd4c0fe099effd3e5098fa7576f18b47c9a69656593"
Jul 16 22:33:52.786693 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.786662    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"]
Jul 16 22:33:52.792402 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:52.792378    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-v8krd"]
Jul 16 22:33:53.750914 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:53.750876    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" path="/var/lib/kubelet/pods/083ea756-54b0-4e5f-8018-55854a1764ff/volumes"
Jul 16 22:33:59.038523 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:59.038490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:59.521793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:59.521762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:59.741438 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:33:59.741401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:02.725797 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.725751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:02.891603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.891571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:02.940814 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.940775    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"]
Jul 16 22:34:02.941121 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941108    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941122    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941129    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941135    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941148    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941154    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941378 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941221    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941378 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941229    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.941378 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.941236    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="083ea756-54b0-4e5f-8018-55854a1764ff" containerName="revoke-keys"
Jul 16 22:34:02.945935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.945913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:02.948926 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.948905    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b3fb3ce5-serving-cert\""
Jul 16 22:34:02.949752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.949732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:34:02.949858 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.949794    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:34:02.949858 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.949810    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:34:02.967204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:02.967181    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"]
Jul 16 22:34:03.005150 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.005061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.005150 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.005094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xzq\" (UniqueName: \"kubernetes.io/projected/9cd092b9-efa5-4818-a7ed-0a71279067aa-kube-api-access-r8xzq\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.005335 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.005194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd092b9-efa5-4818-a7ed-0a71279067aa-openshift-service-ca\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.022980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.022948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:03.106523 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.106490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd092b9-efa5-4818-a7ed-0a71279067aa-openshift-service-ca\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.106716 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.106581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.106716 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.106598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xzq\" (UniqueName: \"kubernetes.io/projected/9cd092b9-efa5-4818-a7ed-0a71279067aa-kube-api-access-r8xzq\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.106792 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:34:03.106722    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-serving-cert: secret "maas-api-e2e-mt-b3fb3ce5-serving-cert" not found
Jul 16 22:34:03.106792 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:34:03.106785    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls podName:9cd092b9-efa5-4818-a7ed-0a71279067aa nodeName:}" failed. No retries permitted until 2026-07-16 22:34:03.606769273 +0000 UTC m=+2826.388667099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls") pod "maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" (UID: "9cd092b9-efa5-4818-a7ed-0a71279067aa") : secret "maas-api-e2e-mt-b3fb3ce5-serving-cert" not found
Jul 16 22:34:03.107225 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.107207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd092b9-efa5-4818-a7ed-0a71279067aa-openshift-service-ca\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.119730 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.119705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xzq\" (UniqueName: \"kubernetes.io/projected/9cd092b9-efa5-4818-a7ed-0a71279067aa-kube-api-access-r8xzq\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.523441 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.523406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:03.610616 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.610575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.613302 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.613275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.857944 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.857850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:03.873883 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.873852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:03.972112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:03.972071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:04.008685 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:04.008657    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"]
Jul 16 22:34:04.011139 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:34:04.011113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd092b9_efa5_4818_a7ed_0a71279067aa.slice/crio-26351592a897df6c5503895a4830a4b4ca12df406f97d35c78c0188554491a3a WatchSource:0}: Error finding container 26351592a897df6c5503895a4830a4b4ca12df406f97d35c78c0188554491a3a: Status 404 returned error can't find the container with id 26351592a897df6c5503895a4830a4b4ca12df406f97d35c78c0188554491a3a
Jul 16 22:34:04.813366 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:04.813335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" event={"ID":"9cd092b9-efa5-4818-a7ed-0a71279067aa","Type":"ContainerStarted","Data":"26351592a897df6c5503895a4830a4b4ca12df406f97d35c78c0188554491a3a"}
Jul 16 22:34:05.818928 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:05.818893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" event={"ID":"9cd092b9-efa5-4818-a7ed-0a71279067aa","Type":"ContainerStarted","Data":"39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20"}
Jul 16 22:34:05.819317 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:05.819003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:05.844538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:05.844488    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" podStartSLOduration=3.097850028 podStartE2EDuration="3.844472846s" totalImagesPullingTime="746.622818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:05.842076709 +0000 UTC m=+2828.623974556" watchObservedRunningTime="2026-07-16 22:34:05.844472846 +0000 UTC m=+2828.626370694"
Jul 16 22:34:07.083233 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:07.083196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:11.829099 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:11.829071    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:34:16.198752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:34:16.198712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:05.502836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:05.502803    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:05.694855 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:05.694817    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:06.518949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:06.518907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:06.619300 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:06.619240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:16.534175 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.534141    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"]
Jul 16 22:35:16.536573 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.534396    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" podUID="9cd092b9-efa5-4818-a7ed-0a71279067aa" containerName="maas-api" containerID="cri-o://39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20" gracePeriod=30
Jul 16 22:35:16.559519 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.559489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:16.781039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.781014    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:35:16.854247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.854155    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd092b9-efa5-4818-a7ed-0a71279067aa-openshift-service-ca\") pod \"9cd092b9-efa5-4818-a7ed-0a71279067aa\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") "
Jul 16 22:35:16.854247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.854210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls\") pod \"9cd092b9-efa5-4818-a7ed-0a71279067aa\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") "
Jul 16 22:35:16.854466 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.854256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8xzq\" (UniqueName: \"kubernetes.io/projected/9cd092b9-efa5-4818-a7ed-0a71279067aa-kube-api-access-r8xzq\") pod \"9cd092b9-efa5-4818-a7ed-0a71279067aa\" (UID: \"9cd092b9-efa5-4818-a7ed-0a71279067aa\") "
Jul 16 22:35:16.854714 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.854623    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd092b9-efa5-4818-a7ed-0a71279067aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9cd092b9-efa5-4818-a7ed-0a71279067aa" (UID: "9cd092b9-efa5-4818-a7ed-0a71279067aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:16.856495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.856473    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9cd092b9-efa5-4818-a7ed-0a71279067aa" (UID: "9cd092b9-efa5-4818-a7ed-0a71279067aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:35:16.856898 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.856872    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd092b9-efa5-4818-a7ed-0a71279067aa-kube-api-access-r8xzq" (OuterVolumeSpecName: "kube-api-access-r8xzq") pod "9cd092b9-efa5-4818-a7ed-0a71279067aa" (UID: "9cd092b9-efa5-4818-a7ed-0a71279067aa"). InnerVolumeSpecName "kube-api-access-r8xzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:35:16.955575 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.955530    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd092b9-efa5-4818-a7ed-0a71279067aa-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:35:16.955575 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.955572    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd092b9-efa5-4818-a7ed-0a71279067aa-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:35:16.955575 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:16.955581    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xzq\" (UniqueName: \"kubernetes.io/projected/9cd092b9-efa5-4818-a7ed-0a71279067aa-kube-api-access-r8xzq\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:35:17.145935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.145831    2562 generic.go:356] "Generic (PLEG): container finished" podID="9cd092b9-efa5-4818-a7ed-0a71279067aa" containerID="39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20" exitCode=0
Jul 16 22:35:17.145935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.145894    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"
Jul 16 22:35:17.146159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.145927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" event={"ID":"9cd092b9-efa5-4818-a7ed-0a71279067aa","Type":"ContainerDied","Data":"39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20"}
Jul 16 22:35:17.146159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.146005    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s" event={"ID":"9cd092b9-efa5-4818-a7ed-0a71279067aa","Type":"ContainerDied","Data":"26351592a897df6c5503895a4830a4b4ca12df406f97d35c78c0188554491a3a"}
Jul 16 22:35:17.146159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.146037    2562 scope.go:117] "RemoveContainer" containerID="39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20"
Jul 16 22:35:17.156252 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.156224    2562 scope.go:117] "RemoveContainer" containerID="39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20"
Jul 16 22:35:17.156518 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:17.156500    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20\": container with ID starting with 39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20 not found: ID does not exist" containerID="39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20"
Jul 16 22:35:17.156630 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.156526    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20"} err="failed to get container status \"39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20\": rpc error: code = NotFound desc = could not find container \"39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20\": container with ID starting with 39081634cadde7a1f16381406752cd931cdbbd7bdd822b9069084895d73cba20 not found: ID does not exist"
Jul 16 22:35:17.188759 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.188733    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"]
Jul 16 22:35:17.196665 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.196644    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3fb3ce5-5456459cb6-8xc4s"]
Jul 16 22:35:17.751979 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:17.751951    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd092b9-efa5-4818-a7ed-0a71279067aa" path="/var/lib/kubelet/pods/9cd092b9-efa5-4818-a7ed-0a71279067aa/volumes"
Jul 16 22:35:21.043468 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:21.043430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:25.552321 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.552289    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"]
Jul 16 22:35:25.552743 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.552671    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd092b9-efa5-4818-a7ed-0a71279067aa" containerName="maas-api"
Jul 16 22:35:25.552743 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.552684    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd092b9-efa5-4818-a7ed-0a71279067aa" containerName="maas-api"
Jul 16 22:35:25.552743 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.552737    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd092b9-efa5-4818-a7ed-0a71279067aa" containerName="maas-api"
Jul 16 22:35:25.557072 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.557057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:25.559778 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.559754    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:35:25.559906 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.559804    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:35:25.572344 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.572322    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"]
Jul 16 22:35:25.632420 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.632391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a0ec05-b124-4858-b77b-8645614823e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6\" (UID: \"47a0ec05-b124-4858-b77b-8645614823e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:25.732925 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.732890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a0ec05-b124-4858-b77b-8645614823e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6\" (UID: \"47a0ec05-b124-4858-b77b-8645614823e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:25.733636 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.733619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a0ec05-b124-4858-b77b-8645614823e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6\" (UID: \"47a0ec05-b124-4858-b77b-8645614823e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:25.867433 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:25.867361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:26.023975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:26.023758    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"]
Jul 16 22:35:26.026295 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:35:26.026267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a0ec05_b124_4858_b77b_8645614823e5.slice/crio-486ee564855faf00668ff8a31e8c7ec40d062c95f9dd11c4472ed105460a5645 WatchSource:0}: Error finding container 486ee564855faf00668ff8a31e8c7ec40d062c95f9dd11c4472ed105460a5645: Status 404 returned error can't find the container with id 486ee564855faf00668ff8a31e8c7ec40d062c95f9dd11c4472ed105460a5645
Jul 16 22:35:26.190688 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:26.190649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" event={"ID":"47a0ec05-b124-4858-b77b-8645614823e5","Type":"ContainerStarted","Data":"486ee564855faf00668ff8a31e8c7ec40d062c95f9dd11c4472ed105460a5645"}
Jul 16 22:35:26.229744 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:26.229710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:27.198022 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:27.197986    2562 generic.go:356] "Generic (PLEG): container finished" podID="47a0ec05-b124-4858-b77b-8645614823e5" containerID="7d7a065c6db4cbb59b6ee4c45d0d1bc0d802e5605d5680f12712de6a15a72e99" exitCode=6
Jul 16 22:35:27.198453 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:27.198067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" event={"ID":"47a0ec05-b124-4858-b77b-8645614823e5","Type":"ContainerDied","Data":"7d7a065c6db4cbb59b6ee4c45d0d1bc0d802e5605d5680f12712de6a15a72e99"}
Jul 16 22:35:27.198453 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:27.198214    2562 scope.go:117] "RemoveContainer" containerID="7d7a065c6db4cbb59b6ee4c45d0d1bc0d802e5605d5680f12712de6a15a72e99"
Jul 16 22:35:28.204628 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:28.204590    2562 generic.go:356] "Generic (PLEG): container finished" podID="47a0ec05-b124-4858-b77b-8645614823e5" containerID="5d8fe2d21178d357e06eaaba0168885fac122dce3d09606a6fed37a9964cc86c" exitCode=6
Jul 16 22:35:28.205052 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:28.204677    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" event={"ID":"47a0ec05-b124-4858-b77b-8645614823e5","Type":"ContainerDied","Data":"5d8fe2d21178d357e06eaaba0168885fac122dce3d09606a6fed37a9964cc86c"}
Jul 16 22:35:28.205052 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:28.204735    2562 scope.go:117] "RemoveContainer" containerID="7d7a065c6db4cbb59b6ee4c45d0d1bc0d802e5605d5680f12712de6a15a72e99"
Jul 16 22:35:28.205052 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:28.204994    2562 scope.go:117] "RemoveContainer" containerID="5d8fe2d21178d357e06eaaba0168885fac122dce3d09606a6fed37a9964cc86c"
Jul 16 22:35:28.205229 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:28.205209    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6_odh-ai-gateway-infra(47a0ec05-b124-4858-b77b-8645614823e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" podUID="47a0ec05-b124-4858-b77b-8645614823e5"
Jul 16 22:35:29.211140 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:29.211111    2562 scope.go:117] "RemoveContainer" containerID="5d8fe2d21178d357e06eaaba0168885fac122dce3d09606a6fed37a9964cc86c"
Jul 16 22:35:29.211581 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:29.211322    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6_odh-ai-gateway-infra(47a0ec05-b124-4858-b77b-8645614823e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" podUID="47a0ec05-b124-4858-b77b-8645614823e5"
Jul 16 22:35:37.124875 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.124831    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"]
Jul 16 22:35:37.127884 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.127860    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:37.138129 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.138103    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"]
Jul 16 22:35:37.235385 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.235351    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588dd15c-dce0-4ea5-a299-b584ac664fa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc\" (UID: \"588dd15c-dce0-4ea5-a299-b584ac664fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:37.336095 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.336059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588dd15c-dce0-4ea5-a299-b584ac664fa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc\" (UID: \"588dd15c-dce0-4ea5-a299-b584ac664fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:37.336741 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.336721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588dd15c-dce0-4ea5-a299-b584ac664fa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc\" (UID: \"588dd15c-dce0-4ea5-a299-b584ac664fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:37.439228 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.439195    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:37.570959 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:37.570926    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"]
Jul 16 22:35:37.576425 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:35:37.576397    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588dd15c_dce0_4ea5_a299_b584ac664fa0.slice/crio-de49d5f768c2deea285f88c200cc326e46ee682e9d32f23be627a02a4ed29bc7 WatchSource:0}: Error finding container de49d5f768c2deea285f88c200cc326e46ee682e9d32f23be627a02a4ed29bc7: Status 404 returned error can't find the container with id de49d5f768c2deea285f88c200cc326e46ee682e9d32f23be627a02a4ed29bc7
Jul 16 22:35:38.251126 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:38.251093    2562 generic.go:356] "Generic (PLEG): container finished" podID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerID="49040778478a86e7736df39fde5137c717949a847e3f887f58a333708c3489a5" exitCode=6
Jul 16 22:35:38.251512 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:38.251179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" event={"ID":"588dd15c-dce0-4ea5-a299-b584ac664fa0","Type":"ContainerDied","Data":"49040778478a86e7736df39fde5137c717949a847e3f887f58a333708c3489a5"}
Jul 16 22:35:38.251512 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:38.251238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" event={"ID":"588dd15c-dce0-4ea5-a299-b584ac664fa0","Type":"ContainerStarted","Data":"de49d5f768c2deea285f88c200cc326e46ee682e9d32f23be627a02a4ed29bc7"}
Jul 16 22:35:38.251512 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:38.251347    2562 scope.go:117] "RemoveContainer" containerID="49040778478a86e7736df39fde5137c717949a847e3f887f58a333708c3489a5"
Jul 16 22:35:39.258166 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:39.258127    2562 generic.go:356] "Generic (PLEG): container finished" podID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerID="23afac58af3add3a42243869cc98cb97f7b903c03ef9ae93ed5f7491e09835f9" exitCode=6
Jul 16 22:35:39.258592 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:39.258194    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" event={"ID":"588dd15c-dce0-4ea5-a299-b584ac664fa0","Type":"ContainerDied","Data":"23afac58af3add3a42243869cc98cb97f7b903c03ef9ae93ed5f7491e09835f9"}
Jul 16 22:35:39.258592 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:39.258280    2562 scope.go:117] "RemoveContainer" containerID="49040778478a86e7736df39fde5137c717949a847e3f887f58a333708c3489a5"
Jul 16 22:35:39.258592 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:39.258429    2562 scope.go:117] "RemoveContainer" containerID="23afac58af3add3a42243869cc98cb97f7b903c03ef9ae93ed5f7491e09835f9"
Jul 16 22:35:39.258723 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:39.258693    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc_odh-ai-gateway-infra(588dd15c-dce0-4ea5-a299-b584ac664fa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0"
Jul 16 22:35:40.263984 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:40.263959    2562 scope.go:117] "RemoveContainer" containerID="23afac58af3add3a42243869cc98cb97f7b903c03ef9ae93ed5f7491e09835f9"
Jul 16 22:35:40.264395 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:40.264147    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc_odh-ai-gateway-infra(588dd15c-dce0-4ea5-a299-b584ac664fa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0"
Jul 16 22:35:41.745744 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:41.745717    2562 scope.go:117] "RemoveContainer" containerID="5d8fe2d21178d357e06eaaba0168885fac122dce3d09606a6fed37a9964cc86c"
Jul 16 22:35:42.274505 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:42.274470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" event={"ID":"47a0ec05-b124-4858-b77b-8645614823e5","Type":"ContainerStarted","Data":"b73432eff1325f9fc3835309bc08cfec34ae9c6194bf32a7f2b7613d21bd26b6"}
Jul 16 22:35:42.294404 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:42.294357    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" podStartSLOduration=17.29434019 podStartE2EDuration="17.29434019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:42.292690143 +0000 UTC m=+2925.074588003" watchObservedRunningTime="2026-07-16 22:35:42.29434019 +0000 UTC m=+2925.076238037"
Jul 16 22:35:43.281251 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:43.281213    2562 generic.go:356] "Generic (PLEG): container finished" podID="47a0ec05-b124-4858-b77b-8645614823e5" containerID="b73432eff1325f9fc3835309bc08cfec34ae9c6194bf32a7f2b7613d21bd26b6" exitCode=6
Jul 16 22:35:43.281728 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:43.281276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" event={"ID":"47a0ec05-b124-4858-b77b-8645614823e5","Type":"ContainerDied","Data":"b73432eff1325f9fc3835309bc08cfec34ae9c6194bf32a7f2b7613d21bd26b6"}
Jul 16 22:35:43.281728 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:43.281323    2562 scope.go:117] "RemoveContainer" containerID="5d8fe2d21178d357e06eaaba0168885fac122dce3d09606a6fed37a9964cc86c"
Jul 16 22:35:43.281728 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:43.281648    2562 scope.go:117] "RemoveContainer" containerID="b73432eff1325f9fc3835309bc08cfec34ae9c6194bf32a7f2b7613d21bd26b6"
Jul 16 22:35:43.281863 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:43.281841    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6_odh-ai-gateway-infra(47a0ec05-b124-4858-b77b-8645614823e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" podUID="47a0ec05-b124-4858-b77b-8645614823e5"
Jul 16 22:35:43.316477 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:43.316448    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"]
Jul 16 22:35:44.418026 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:44.417997    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:44.493223 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:44.493191    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a0ec05-b124-4858-b77b-8645614823e5-maas-api-service-ca\") pod \"47a0ec05-b124-4858-b77b-8645614823e5\" (UID: \"47a0ec05-b124-4858-b77b-8645614823e5\") "
Jul 16 22:35:44.493570 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:44.493529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a0ec05-b124-4858-b77b-8645614823e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47a0ec05-b124-4858-b77b-8645614823e5" (UID: "47a0ec05-b124-4858-b77b-8645614823e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:44.593999 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:44.593915    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a0ec05-b124-4858-b77b-8645614823e5-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:35:45.292464 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:45.292432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6" event={"ID":"47a0ec05-b124-4858-b77b-8645614823e5","Type":"ContainerDied","Data":"486ee564855faf00668ff8a31e8c7ec40d062c95f9dd11c4472ed105460a5645"}
Jul 16 22:35:45.292464 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:45.292466    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"
Jul 16 22:35:45.292739 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:45.292490    2562 scope.go:117] "RemoveContainer" containerID="b73432eff1325f9fc3835309bc08cfec34ae9c6194bf32a7f2b7613d21bd26b6"
Jul 16 22:35:45.324740 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:45.324712    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"]
Jul 16 22:35:45.330192 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:45.330166    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-j84b6"]
Jul 16 22:35:45.750434 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:45.750406    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a0ec05-b124-4858-b77b-8645614823e5" path="/var/lib/kubelet/pods/47a0ec05-b124-4858-b77b-8645614823e5/volumes"
Jul 16 22:35:50.745781 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:50.745753    2562 scope.go:117] "RemoveContainer" containerID="23afac58af3add3a42243869cc98cb97f7b903c03ef9ae93ed5f7491e09835f9"
Jul 16 22:35:51.324403 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:51.324317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" event={"ID":"588dd15c-dce0-4ea5-a299-b584ac664fa0","Type":"ContainerStarted","Data":"f25224fcaef8fcc17c4070297925a51e45d743959709b9aab25ea7d3947ce275"}
Jul 16 22:35:51.342391 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:51.342338    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" podStartSLOduration=14.342320903 podStartE2EDuration="14.342320903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:51.341139992 +0000 UTC m=+2934.123037840" watchObservedRunningTime="2026-07-16 22:35:51.342320903 +0000 UTC m=+2934.124218751"
Jul 16 22:35:52.330772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:52.330739    2562 generic.go:356] "Generic (PLEG): container finished" podID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerID="f25224fcaef8fcc17c4070297925a51e45d743959709b9aab25ea7d3947ce275" exitCode=6
Jul 16 22:35:52.331166 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:52.330815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" event={"ID":"588dd15c-dce0-4ea5-a299-b584ac664fa0","Type":"ContainerDied","Data":"f25224fcaef8fcc17c4070297925a51e45d743959709b9aab25ea7d3947ce275"}
Jul 16 22:35:52.331166 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:52.330883    2562 scope.go:117] "RemoveContainer" containerID="23afac58af3add3a42243869cc98cb97f7b903c03ef9ae93ed5f7491e09835f9"
Jul 16 22:35:52.331283 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:52.331183    2562 scope.go:117] "RemoveContainer" containerID="f25224fcaef8fcc17c4070297925a51e45d743959709b9aab25ea7d3947ce275"
Jul 16 22:35:52.331421 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:35:52.331400    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc_odh-ai-gateway-infra(588dd15c-dce0-4ea5-a299-b584ac664fa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0"
Jul 16 22:35:52.356527 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:52.356494    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"]
Jul 16 22:35:53.468510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:53.468489    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:53.565683 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:53.565648    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588dd15c-dce0-4ea5-a299-b584ac664fa0-maas-api-service-ca\") pod \"588dd15c-dce0-4ea5-a299-b584ac664fa0\" (UID: \"588dd15c-dce0-4ea5-a299-b584ac664fa0\") "
Jul 16 22:35:53.566023 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:53.565997    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588dd15c-dce0-4ea5-a299-b584ac664fa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "588dd15c-dce0-4ea5-a299-b584ac664fa0" (UID: "588dd15c-dce0-4ea5-a299-b584ac664fa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:53.666168 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:53.666137    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/588dd15c-dce0-4ea5-a299-b584ac664fa0-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:35:54.343437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:54.343405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc" event={"ID":"588dd15c-dce0-4ea5-a299-b584ac664fa0","Type":"ContainerDied","Data":"de49d5f768c2deea285f88c200cc326e46ee682e9d32f23be627a02a4ed29bc7"}
Jul 16 22:35:54.343437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:54.343447    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"
Jul 16 22:35:54.343705 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:54.343467    2562 scope.go:117] "RemoveContainer" containerID="f25224fcaef8fcc17c4070297925a51e45d743959709b9aab25ea7d3947ce275"
Jul 16 22:35:54.362756 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:54.362730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"]
Jul 16 22:35:54.365233 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:54.365213    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k5zgc"]
Jul 16 22:35:55.750659 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:35:55.750628    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" path="/var/lib/kubelet/pods/588dd15c-dce0-4ea5-a299-b584ac664fa0/volumes"
Jul 16 22:36:02.790572 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:02.790520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:04.087128 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:04.087097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:10.042140 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:10.042095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:15.613104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613069    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"]
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613390    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613400    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613409    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613415    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613427    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613433    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613446    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613451    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613458    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613464    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613511    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613519    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613527    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:15.613838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613532    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613537    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.613838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.613542    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a0ec05-b124-4858-b77b-8645614823e5" containerName="revoke-keys"
Jul 16 22:36:15.616558 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.616527    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:15.619691 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.619669    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:36:15.619691 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.619688    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:36:15.625060 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.625039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"]
Jul 16 22:36:15.759973 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.759938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025fcf13-33a0-4acb-b931-f42dfe4d0ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp\" (UID: \"025fcf13-33a0-4acb-b931-f42dfe4d0ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:15.861341 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.861304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025fcf13-33a0-4acb-b931-f42dfe4d0ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp\" (UID: \"025fcf13-33a0-4acb-b931-f42dfe4d0ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:15.861991 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.861928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025fcf13-33a0-4acb-b931-f42dfe4d0ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp\" (UID: \"025fcf13-33a0-4acb-b931-f42dfe4d0ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:15.928187 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:15.928160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:16.061023 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:16.061000    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"]
Jul 16 22:36:16.063590 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:36:16.063561    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025fcf13_33a0_4acb_b931_f42dfe4d0ea5.slice/crio-a5fd1604c838c1bec57376275a908bdda68312389b0e25dbaa2bd3cd58624eac WatchSource:0}: Error finding container a5fd1604c838c1bec57376275a908bdda68312389b0e25dbaa2bd3cd58624eac: Status 404 returned error can't find the container with id a5fd1604c838c1bec57376275a908bdda68312389b0e25dbaa2bd3cd58624eac
Jul 16 22:36:16.443898 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:16.443864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" event={"ID":"025fcf13-33a0-4acb-b931-f42dfe4d0ea5","Type":"ContainerStarted","Data":"65e7bfe0ed6bc461b87ac189b184cd398e5c770b7d6b3d7cf065d620c2811d1b"}
Jul 16 22:36:16.444105 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:16.443919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" event={"ID":"025fcf13-33a0-4acb-b931-f42dfe4d0ea5","Type":"ContainerStarted","Data":"a5fd1604c838c1bec57376275a908bdda68312389b0e25dbaa2bd3cd58624eac"}
Jul 16 22:36:16.460735 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:16.460686    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" podStartSLOduration=1.460643744 podStartE2EDuration="1.460643744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:16.459852189 +0000 UTC m=+2959.241750037" watchObservedRunningTime="2026-07-16 22:36:16.460643744 +0000 UTC m=+2959.242541626"
Jul 16 22:36:17.449982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:17.449947    2562 generic.go:356] "Generic (PLEG): container finished" podID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerID="65e7bfe0ed6bc461b87ac189b184cd398e5c770b7d6b3d7cf065d620c2811d1b" exitCode=6
Jul 16 22:36:17.450438 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:17.450031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" event={"ID":"025fcf13-33a0-4acb-b931-f42dfe4d0ea5","Type":"ContainerDied","Data":"65e7bfe0ed6bc461b87ac189b184cd398e5c770b7d6b3d7cf065d620c2811d1b"}
Jul 16 22:36:17.450438 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:17.450257    2562 scope.go:117] "RemoveContainer" containerID="65e7bfe0ed6bc461b87ac189b184cd398e5c770b7d6b3d7cf065d620c2811d1b"
Jul 16 22:36:18.456876 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:18.456838    2562 generic.go:356] "Generic (PLEG): container finished" podID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerID="c49aaf144be02675032305dafb3bda881e2baa119106f45ae9d1d41c6a0e2f34" exitCode=6
Jul 16 22:36:18.457267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:18.456930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" event={"ID":"025fcf13-33a0-4acb-b931-f42dfe4d0ea5","Type":"ContainerDied","Data":"c49aaf144be02675032305dafb3bda881e2baa119106f45ae9d1d41c6a0e2f34"}
Jul 16 22:36:18.457267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:18.456996    2562 scope.go:117] "RemoveContainer" containerID="65e7bfe0ed6bc461b87ac189b184cd398e5c770b7d6b3d7cf065d620c2811d1b"
Jul 16 22:36:18.457267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:18.457143    2562 scope.go:117] "RemoveContainer" containerID="c49aaf144be02675032305dafb3bda881e2baa119106f45ae9d1d41c6a0e2f34"
Jul 16 22:36:18.457429 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:18.457375    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp_odh-ai-gateway-infra(025fcf13-33a0-4acb-b931-f42dfe4d0ea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5"
Jul 16 22:36:18.471457 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:18.471432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:19.463662 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:19.463637    2562 scope.go:117] "RemoveContainer" containerID="c49aaf144be02675032305dafb3bda881e2baa119106f45ae9d1d41c6a0e2f34"
Jul 16 22:36:19.464112 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:19.463934    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp_odh-ai-gateway-infra(025fcf13-33a0-4acb-b931-f42dfe4d0ea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5"
Jul 16 22:36:24.789708 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.789675    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"]
Jul 16 22:36:24.790114 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.790016    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:24.790114 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.790028    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="588dd15c-dce0-4ea5-a299-b584ac664fa0" containerName="revoke-keys"
Jul 16 22:36:24.794174 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.794158    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:24.800601 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.800576    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"]
Jul 16 22:36:24.923486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.923446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:24.939665 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.939628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba303f06-c776-484a-9394-9a134cca844e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5\" (UID: \"ba303f06-c776-484a-9394-9a134cca844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:24.992194 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:24.992160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:25.040545 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.040455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba303f06-c776-484a-9394-9a134cca844e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5\" (UID: \"ba303f06-c776-484a-9394-9a134cca844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:25.041109 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.041089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba303f06-c776-484a-9394-9a134cca844e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5\" (UID: \"ba303f06-c776-484a-9394-9a134cca844e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:25.106389 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.106359    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:25.239811 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.239773    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"]
Jul 16 22:36:25.242170 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:36:25.242147    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba303f06_c776_484a_9394_9a134cca844e.slice/crio-88f2b7988f86167ed0969c4c7c8fd8d315faa82cb8977e700ed67e296e3c2474 WatchSource:0}: Error finding container 88f2b7988f86167ed0969c4c7c8fd8d315faa82cb8977e700ed67e296e3c2474: Status 404 returned error can't find the container with id 88f2b7988f86167ed0969c4c7c8fd8d315faa82cb8977e700ed67e296e3c2474
Jul 16 22:36:25.491469 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.491437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" event={"ID":"ba303f06-c776-484a-9394-9a134cca844e","Type":"ContainerStarted","Data":"aa1fe7936d0b299a5a8335ad8e8ccff3c72da0b1e67dfa22ac88fef8a713b7b2"}
Jul 16 22:36:25.491615 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.491494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" event={"ID":"ba303f06-c776-484a-9394-9a134cca844e","Type":"ContainerStarted","Data":"88f2b7988f86167ed0969c4c7c8fd8d315faa82cb8977e700ed67e296e3c2474"}
Jul 16 22:36:25.507497 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:25.507453    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" podStartSLOduration=1.507438128 podStartE2EDuration="1.507438128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:25.506417454 +0000 UTC m=+2968.288315305" watchObservedRunningTime="2026-07-16 22:36:25.507438128 +0000 UTC m=+2968.289335976"
Jul 16 22:36:26.497935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:26.497900    2562 generic.go:356] "Generic (PLEG): container finished" podID="ba303f06-c776-484a-9394-9a134cca844e" containerID="aa1fe7936d0b299a5a8335ad8e8ccff3c72da0b1e67dfa22ac88fef8a713b7b2" exitCode=6
Jul 16 22:36:26.498349 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:26.497995    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" event={"ID":"ba303f06-c776-484a-9394-9a134cca844e","Type":"ContainerDied","Data":"aa1fe7936d0b299a5a8335ad8e8ccff3c72da0b1e67dfa22ac88fef8a713b7b2"}
Jul 16 22:36:26.498349 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:26.498178    2562 scope.go:117] "RemoveContainer" containerID="aa1fe7936d0b299a5a8335ad8e8ccff3c72da0b1e67dfa22ac88fef8a713b7b2"
Jul 16 22:36:27.167539 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.167510    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"]
Jul 16 22:36:27.170798 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.170779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:27.178245 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.178217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"]
Jul 16 22:36:27.259919 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.259890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb\" (UID: \"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:27.361354 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.361314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb\" (UID: \"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:27.361984 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.361962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb\" (UID: \"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:27.504857 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.504817    2562 generic.go:356] "Generic (PLEG): container finished" podID="ba303f06-c776-484a-9394-9a134cca844e" containerID="4faabd27a48ce8fa29cb244b30de0f0f258bb7e2529e6b6baa8e5050e801a816" exitCode=6
Jul 16 22:36:27.505267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.504904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" event={"ID":"ba303f06-c776-484a-9394-9a134cca844e","Type":"ContainerDied","Data":"4faabd27a48ce8fa29cb244b30de0f0f258bb7e2529e6b6baa8e5050e801a816"}
Jul 16 22:36:27.505267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.504969    2562 scope.go:117] "RemoveContainer" containerID="aa1fe7936d0b299a5a8335ad8e8ccff3c72da0b1e67dfa22ac88fef8a713b7b2"
Jul 16 22:36:27.505267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.505224    2562 scope.go:117] "RemoveContainer" containerID="4faabd27a48ce8fa29cb244b30de0f0f258bb7e2529e6b6baa8e5050e801a816"
Jul 16 22:36:27.505462 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:27.505441    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5_odh-ai-gateway-infra(ba303f06-c776-484a-9394-9a134cca844e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" podUID="ba303f06-c776-484a-9394-9a134cca844e"
Jul 16 22:36:27.517927 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.517912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:27.652777 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:27.652746    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"]
Jul 16 22:36:27.654428 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:36:27.654393    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e3046b_c8c9_44ff_9d80_b6d7ccc8a60f.slice/crio-88e189b4ec1d7c3578bdc3482a5c4385389190dd31de63f3d4cc15ca108bb46b WatchSource:0}: Error finding container 88e189b4ec1d7c3578bdc3482a5c4385389190dd31de63f3d4cc15ca108bb46b: Status 404 returned error can't find the container with id 88e189b4ec1d7c3578bdc3482a5c4385389190dd31de63f3d4cc15ca108bb46b
Jul 16 22:36:28.511227 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:28.511143    2562 scope.go:117] "RemoveContainer" containerID="4faabd27a48ce8fa29cb244b30de0f0f258bb7e2529e6b6baa8e5050e801a816"
Jul 16 22:36:28.511645 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:28.511360    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5_odh-ai-gateway-infra(ba303f06-c776-484a-9394-9a134cca844e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" podUID="ba303f06-c776-484a-9394-9a134cca844e"
Jul 16 22:36:28.512441 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:28.512416    2562 generic.go:356] "Generic (PLEG): container finished" podID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerID="695d086717d0c0da853adacb740047ab33cfc65f6db249d905935342de740e7d" exitCode=6
Jul 16 22:36:28.512534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:28.512452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" event={"ID":"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f","Type":"ContainerDied","Data":"695d086717d0c0da853adacb740047ab33cfc65f6db249d905935342de740e7d"}
Jul 16 22:36:28.512534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:28.512526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" event={"ID":"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f","Type":"ContainerStarted","Data":"88e189b4ec1d7c3578bdc3482a5c4385389190dd31de63f3d4cc15ca108bb46b"}
Jul 16 22:36:28.512703 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:28.512692    2562 scope.go:117] "RemoveContainer" containerID="695d086717d0c0da853adacb740047ab33cfc65f6db249d905935342de740e7d"
Jul 16 22:36:29.519856 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:29.519821    2562 generic.go:356] "Generic (PLEG): container finished" podID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerID="fccce2349c72f4815451ac2aed03b11d2b4c90136bce579adb6747727a1b7c5a" exitCode=6
Jul 16 22:36:29.520267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:29.519903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" event={"ID":"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f","Type":"ContainerDied","Data":"fccce2349c72f4815451ac2aed03b11d2b4c90136bce579adb6747727a1b7c5a"}
Jul 16 22:36:29.520267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:29.519978    2562 scope.go:117] "RemoveContainer" containerID="695d086717d0c0da853adacb740047ab33cfc65f6db249d905935342de740e7d"
Jul 16 22:36:29.520267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:29.520134    2562 scope.go:117] "RemoveContainer" containerID="fccce2349c72f4815451ac2aed03b11d2b4c90136bce579adb6747727a1b7c5a"
Jul 16 22:36:29.520388 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:29.520351    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb_odh-ai-gateway-infra(05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f"
Jul 16 22:36:30.526632 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:30.526604    2562 scope.go:117] "RemoveContainer" containerID="fccce2349c72f4815451ac2aed03b11d2b4c90136bce579adb6747727a1b7c5a"
Jul 16 22:36:30.527041 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:30.526776    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb_odh-ai-gateway-infra(05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f"
Jul 16 22:36:30.745789 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:30.745758    2562 scope.go:117] "RemoveContainer" containerID="c49aaf144be02675032305dafb3bda881e2baa119106f45ae9d1d41c6a0e2f34"
Jul 16 22:36:31.532839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:31.532803    2562 generic.go:356] "Generic (PLEG): container finished" podID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerID="5c055c58080c5d63ada3e6752ec55055e6686fdf62c9d475b56e3a79946156c6" exitCode=6
Jul 16 22:36:31.533257 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:31.532843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" event={"ID":"025fcf13-33a0-4acb-b931-f42dfe4d0ea5","Type":"ContainerDied","Data":"5c055c58080c5d63ada3e6752ec55055e6686fdf62c9d475b56e3a79946156c6"}
Jul 16 22:36:31.533257 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:31.532918    2562 scope.go:117] "RemoveContainer" containerID="c49aaf144be02675032305dafb3bda881e2baa119106f45ae9d1d41c6a0e2f34"
Jul 16 22:36:31.533376 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:31.533282    2562 scope.go:117] "RemoveContainer" containerID="5c055c58080c5d63ada3e6752ec55055e6686fdf62c9d475b56e3a79946156c6"
Jul 16 22:36:31.533520 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:31.533491    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp_odh-ai-gateway-infra(025fcf13-33a0-4acb-b931-f42dfe4d0ea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5"
Jul 16 22:36:32.561911 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:32.561875    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"]
Jul 16 22:36:32.694636 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:32.694614    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:32.815795 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:32.815702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025fcf13-33a0-4acb-b931-f42dfe4d0ea5-maas-api-service-ca\") pod \"025fcf13-33a0-4acb-b931-f42dfe4d0ea5\" (UID: \"025fcf13-33a0-4acb-b931-f42dfe4d0ea5\") "
Jul 16 22:36:32.816102 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:32.816075    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025fcf13-33a0-4acb-b931-f42dfe4d0ea5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "025fcf13-33a0-4acb-b931-f42dfe4d0ea5" (UID: "025fcf13-33a0-4acb-b931-f42dfe4d0ea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:32.916876 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:32.916835    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025fcf13-33a0-4acb-b931-f42dfe4d0ea5-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.424174 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.424134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:33.545732 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.545704    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"
Jul 16 22:36:33.545910 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.545703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp" event={"ID":"025fcf13-33a0-4acb-b931-f42dfe4d0ea5","Type":"ContainerDied","Data":"a5fd1604c838c1bec57376275a908bdda68312389b0e25dbaa2bd3cd58624eac"}
Jul 16 22:36:33.545910 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.545883    2562 scope.go:117] "RemoveContainer" containerID="5c055c58080c5d63ada3e6752ec55055e6686fdf62c9d475b56e3a79946156c6"
Jul 16 22:36:33.580769 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.580734    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"]
Jul 16 22:36:33.589890 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.589866    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2fvrp"]
Jul 16 22:36:33.661597 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.661565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:33.750878 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:33.750798    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" path="/var/lib/kubelet/pods/025fcf13-33a0-4acb-b931-f42dfe4d0ea5/volumes"
Jul 16 22:36:34.733663 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:34.733631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:34.827854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:34.827824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:34.924666 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:34.924630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:35.658292 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:35.658261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:35.756706 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:35.756678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:35.857532 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:35.857504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:37.219031 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:37.218996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:42.745116 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:42.745082    2562 scope.go:117] "RemoveContainer" containerID="4faabd27a48ce8fa29cb244b30de0f0f258bb7e2529e6b6baa8e5050e801a816"
Jul 16 22:36:43.596228 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:43.596193    2562 generic.go:356] "Generic (PLEG): container finished" podID="ba303f06-c776-484a-9394-9a134cca844e" containerID="6d195ad7dfea7ca593050a838c574475ba24326fa9fc80b5cff68164115675c3" exitCode=6
Jul 16 22:36:43.596401 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:43.596268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" event={"ID":"ba303f06-c776-484a-9394-9a134cca844e","Type":"ContainerDied","Data":"6d195ad7dfea7ca593050a838c574475ba24326fa9fc80b5cff68164115675c3"}
Jul 16 22:36:43.596401 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:43.596336    2562 scope.go:117] "RemoveContainer" containerID="4faabd27a48ce8fa29cb244b30de0f0f258bb7e2529e6b6baa8e5050e801a816"
Jul 16 22:36:43.596694 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:43.596669    2562 scope.go:117] "RemoveContainer" containerID="6d195ad7dfea7ca593050a838c574475ba24326fa9fc80b5cff68164115675c3"
Jul 16 22:36:43.596931 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:43.596897    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5_odh-ai-gateway-infra(ba303f06-c776-484a-9394-9a134cca844e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" podUID="ba303f06-c776-484a-9394-9a134cca844e"
Jul 16 22:36:44.652610 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:44.652574    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"]
Jul 16 22:36:44.792935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:44.792908    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:44.820584 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:44.820555    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba303f06-c776-484a-9394-9a134cca844e-maas-api-service-ca\") pod \"ba303f06-c776-484a-9394-9a134cca844e\" (UID: \"ba303f06-c776-484a-9394-9a134cca844e\") "
Jul 16 22:36:44.820855 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:44.820831    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba303f06-c776-484a-9394-9a134cca844e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba303f06-c776-484a-9394-9a134cca844e" (UID: "ba303f06-c776-484a-9394-9a134cca844e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:44.922114 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:44.922038    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba303f06-c776-484a-9394-9a134cca844e-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:36:44.959663 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:44.959632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:45.062317 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.062282    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:45.609639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.609609    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"
Jul 16 22:36:45.609809 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.609654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5" event={"ID":"ba303f06-c776-484a-9394-9a134cca844e","Type":"ContainerDied","Data":"88f2b7988f86167ed0969c4c7c8fd8d315faa82cb8977e700ed67e296e3c2474"}
Jul 16 22:36:45.609809 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.609719    2562 scope.go:117] "RemoveContainer" containerID="6d195ad7dfea7ca593050a838c574475ba24326fa9fc80b5cff68164115675c3"
Jul 16 22:36:45.637211 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.637184    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"]
Jul 16 22:36:45.641061 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.641039    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zcqr5"]
Jul 16 22:36:45.668472 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.668441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:45.745139 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.745110    2562 scope.go:117] "RemoveContainer" containerID="fccce2349c72f4815451ac2aed03b11d2b4c90136bce579adb6747727a1b7c5a"
Jul 16 22:36:45.751728 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:45.751697    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba303f06-c776-484a-9394-9a134cca844e" path="/var/lib/kubelet/pods/ba303f06-c776-484a-9394-9a134cca844e/volumes"
Jul 16 22:36:46.617337 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:46.617251    2562 generic.go:356] "Generic (PLEG): container finished" podID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerID="ba84f3b346c33184593995b02fd2471c50e6311b1e8a9fbc58867970178b6baa" exitCode=6
Jul 16 22:36:46.617337 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:46.617294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" event={"ID":"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f","Type":"ContainerDied","Data":"ba84f3b346c33184593995b02fd2471c50e6311b1e8a9fbc58867970178b6baa"}
Jul 16 22:36:46.617571 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:46.617353    2562 scope.go:117] "RemoveContainer" containerID="fccce2349c72f4815451ac2aed03b11d2b4c90136bce579adb6747727a1b7c5a"
Jul 16 22:36:46.617679 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:46.617657    2562 scope.go:117] "RemoveContainer" containerID="ba84f3b346c33184593995b02fd2471c50e6311b1e8a9fbc58867970178b6baa"
Jul 16 22:36:46.617927 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:36:46.617903    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb_odh-ai-gateway-infra(05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f"
Jul 16 22:36:47.651927 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:47.651895    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"]
Jul 16 22:36:47.785892 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:47.785867    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:47.850509 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:47.850477    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f-maas-api-service-ca\") pod \"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f\" (UID: \"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f\") "
Jul 16 22:36:47.850852 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:47.850828    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" (UID: "05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:47.952079 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:47.951991    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:36:48.628770 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:48.628743    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"
Jul 16 22:36:48.628943 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:48.628739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb" event={"ID":"05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f","Type":"ContainerDied","Data":"88e189b4ec1d7c3578bdc3482a5c4385389190dd31de63f3d4cc15ca108bb46b"}
Jul 16 22:36:48.628943 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:48.628877    2562 scope.go:117] "RemoveContainer" containerID="ba84f3b346c33184593995b02fd2471c50e6311b1e8a9fbc58867970178b6baa"
Jul 16 22:36:48.657395 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:48.657372    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"]
Jul 16 22:36:48.662705 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:48.662681    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6hvsb"]
Jul 16 22:36:49.751869 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:49.751829    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" path="/var/lib/kubelet/pods/05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f/volumes"
Jul 16 22:36:52.660718 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:52.660685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:52.778282 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:52.778248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:52.885702 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:52.885670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:56.906870 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:36:56.906837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:03.411859 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:03.411828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:03.767407 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:03.767299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:24.758992 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:24.758957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:31.142836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:31.142802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:31.306787 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:31.306749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:31.494998 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:31.494967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:31.856225 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:31.856121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:32.822532 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:32.822497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:34.720015 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.719977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:34.797664 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.797636    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"]
Jul 16 22:37:34.798025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798008    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798028    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798048    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798056    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798065    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798073    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798092    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.798112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798108    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798116    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798128    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798136    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798237    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798250    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798260    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798270    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798278    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798288    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.798396 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.798297    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:34.801943 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.801926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.806231 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.806210    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3a3e4b-serving-cert\""
Jul 16 22:37:34.806801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.806783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:37:34.806951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.806933    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:37:34.807034 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.806933    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:37:34.817392 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.817365    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:34.822379 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.822355    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"]
Jul 16 22:37:34.850473 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.850430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.850662 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.850517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860a3dbc-d2e0-4163-b010-786c96122bf8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.850740 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.850688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpjz\" (UniqueName: \"kubernetes.io/projected/860a3dbc-d2e0-4163-b010-786c96122bf8-kube-api-access-6jpjz\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.905558 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.905526    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:34.951239 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.951206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.951392 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.951252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860a3dbc-d2e0-4163-b010-786c96122bf8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.951392 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:37:34.951364    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-serving-cert: secret "maas-api-e2e-inf-a-3a3e4b-serving-cert" not found
Jul 16 22:37:34.951477 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:37:34.951445    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls podName:860a3dbc-d2e0-4163-b010-786c96122bf8 nodeName:}" failed. No retries permitted until 2026-07-16 22:37:35.451424059 +0000 UTC m=+3038.233321884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls") pod "maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" (UID: "860a3dbc-d2e0-4163-b010-786c96122bf8") : secret "maas-api-e2e-inf-a-3a3e4b-serving-cert" not found
Jul 16 22:37:34.951533 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.951498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpjz\" (UniqueName: \"kubernetes.io/projected/860a3dbc-d2e0-4163-b010-786c96122bf8-kube-api-access-6jpjz\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.952143 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.952123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860a3dbc-d2e0-4163-b010-786c96122bf8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:34.963871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:34.963837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpjz\" (UniqueName: \"kubernetes.io/projected/860a3dbc-d2e0-4163-b010-786c96122bf8-kube-api-access-6jpjz\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:35.359104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:35.359073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:35.455905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:35.455869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:35.458313 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:35.458279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:35.458912 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:35.458892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:35.720233 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:35.720205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:35.862326 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:35.862298    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"]
Jul 16 22:37:35.865130 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:37:35.865089    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860a3dbc_d2e0_4163_b010_786c96122bf8.slice/crio-577f99b44267e518eb68ac3aca8639e7dbae26016effb569c85a4c5d349b9c99 WatchSource:0}: Error finding container 577f99b44267e518eb68ac3aca8639e7dbae26016effb569c85a4c5d349b9c99: Status 404 returned error can't find the container with id 577f99b44267e518eb68ac3aca8639e7dbae26016effb569c85a4c5d349b9c99
Jul 16 22:37:36.845268 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:36.845160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" event={"ID":"860a3dbc-d2e0-4163-b010-786c96122bf8","Type":"ContainerStarted","Data":"4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e"}
Jul 16 22:37:36.845268 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:36.845235    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:36.845268 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:36.845263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" event={"ID":"860a3dbc-d2e0-4163-b010-786c96122bf8","Type":"ContainerStarted","Data":"577f99b44267e518eb68ac3aca8639e7dbae26016effb569c85a4c5d349b9c99"}
Jul 16 22:37:36.868414 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:36.868365    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" podStartSLOduration=2.3049887399999998 podStartE2EDuration="2.868349604s" totalImagesPullingTime="563.360864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:36.867949898 +0000 UTC m=+3039.649847749" watchObservedRunningTime="2026-07-16 22:37:36.868349604 +0000 UTC m=+3039.650247468"
Jul 16 22:37:39.440980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:39.440947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:39.693087 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:39.692998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:40.051685 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:40.051601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:40.417383 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:40.417329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:42.756834 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.756802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:42.832936 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.832888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:42.856787 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.856757    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:37:42.939094 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939063    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"]
Jul 16 22:37:42.939398 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939384    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:42.939398 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939398    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:42.939534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939406    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:42.939534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939411    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="025fcf13-33a0-4acb-b931-f42dfe4d0ea5" containerName="revoke-keys"
Jul 16 22:37:42.939534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939423    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:42.939534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939428    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:42.939534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939489    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba303f06-c776-484a-9394-9a134cca844e" containerName="revoke-keys"
Jul 16 22:37:42.939534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.939506    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e3046b-c8c9-44ff-9d80-b6d7ccc8a60f" containerName="revoke-keys"
Jul 16 22:37:42.942776 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.942752    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:42.945405 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.945383    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2fe960-serving-cert\""
Jul 16 22:37:42.961676 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.961645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"]
Jul 16 22:37:42.984374 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:42.984345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:43.020831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.020745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.020831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.020783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560a29b-2e14-4ee2-a150-75d7e7624a9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.020831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.020806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwc89\" (UniqueName: \"kubernetes.io/projected/4560a29b-2e14-4ee2-a150-75d7e7624a9c-kube-api-access-bwc89\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.122247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.122208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.122247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.122251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560a29b-2e14-4ee2-a150-75d7e7624a9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.122518 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.122273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwc89\" (UniqueName: \"kubernetes.io/projected/4560a29b-2e14-4ee2-a150-75d7e7624a9c-kube-api-access-bwc89\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.122518 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:37:43.122359    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-serving-cert: secret "maas-api-e2e-inf-b-2fe960-serving-cert" not found
Jul 16 22:37:43.122518 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:37:43.122424    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls podName:4560a29b-2e14-4ee2-a150-75d7e7624a9c nodeName:}" failed. No retries permitted until 2026-07-16 22:37:43.622405926 +0000 UTC m=+3046.404303752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls") pod "maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" (UID: "4560a29b-2e14-4ee2-a150-75d7e7624a9c") : secret "maas-api-e2e-inf-b-2fe960-serving-cert" not found
Jul 16 22:37:43.123017 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.122996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560a29b-2e14-4ee2-a150-75d7e7624a9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.131805 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.131781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwc89\" (UniqueName: \"kubernetes.io/projected/4560a29b-2e14-4ee2-a150-75d7e7624a9c-kube-api-access-bwc89\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.234372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.234341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:43.552202 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.552162    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:43.626951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.626908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.629568 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.629533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.662232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.662198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:43.855017 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.854921    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:43.996484 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:43.996454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"]
Jul 16 22:37:43.998609 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:37:43.998578    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4560a29b_2e14_4ee2_a150_75d7e7624a9c.slice/crio-a4277f2eeb1859c194d482e816795f63480a3acaa9ee6fea9504643178fbf8ee WatchSource:0}: Error finding container a4277f2eeb1859c194d482e816795f63480a3acaa9ee6fea9504643178fbf8ee: Status 404 returned error can't find the container with id a4277f2eeb1859c194d482e816795f63480a3acaa9ee6fea9504643178fbf8ee
Jul 16 22:37:44.882795 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:44.882756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" event={"ID":"4560a29b-2e14-4ee2-a150-75d7e7624a9c","Type":"ContainerStarted","Data":"84ded6249a3fbcda62f4bb154a105a59d0ed1d58643b2d60ef29215dc515b166"}
Jul 16 22:37:44.883199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:44.882828    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:44.883199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:44.882854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" event={"ID":"4560a29b-2e14-4ee2-a150-75d7e7624a9c","Type":"ContainerStarted","Data":"a4277f2eeb1859c194d482e816795f63480a3acaa9ee6fea9504643178fbf8ee"}
Jul 16 22:37:47.261717 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:47.261680    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:50.893367 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:50.893287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:37:50.918442 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:50.918395    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" podStartSLOduration=8.601246857 podStartE2EDuration="8.91837932s" totalImagesPullingTime="317.132463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:44.904285116 +0000 UTC m=+3047.686182963" watchObservedRunningTime="2026-07-16 22:37:50.91837932 +0000 UTC m=+3053.700277167"
Jul 16 22:37:55.769816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.769777    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"]
Jul 16 22:37:55.773390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.773371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:37:55.776472 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.776452    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:37:55.786302 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.786277    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"]
Jul 16 22:37:55.826600 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.826568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx\" (UID: \"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:37:55.927898 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.927860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx\" (UID: \"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:37:55.928476 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:55.928456    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx\" (UID: \"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:37:56.083239 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.083159    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:37:56.229172 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.229134    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"]
Jul 16 22:37:56.230708 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:37:56.230685    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e3ef20_b1b0_4ee3_a99c_7b2d0ac160c7.slice/crio-f89166bc59b50e577f3d086be2dfbd3fe0608d5279da4973ed17a0247733e660 WatchSource:0}: Error finding container f89166bc59b50e577f3d086be2dfbd3fe0608d5279da4973ed17a0247733e660: Status 404 returned error can't find the container with id f89166bc59b50e577f3d086be2dfbd3fe0608d5279da4973ed17a0247733e660
Jul 16 22:37:56.386780 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.386699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:56.484059 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.484021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:56.585792 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.585761    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:56.941461 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.941423    2562 generic.go:356] "Generic (PLEG): container finished" podID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerID="9045c19c40e2ddf86ca80693ac062205ae40e26177d4495623c992c076b1c140" exitCode=6
Jul 16 22:37:56.941871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.941504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" event={"ID":"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7","Type":"ContainerDied","Data":"9045c19c40e2ddf86ca80693ac062205ae40e26177d4495623c992c076b1c140"}
Jul 16 22:37:56.941871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.941585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" event={"ID":"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7","Type":"ContainerStarted","Data":"f89166bc59b50e577f3d086be2dfbd3fe0608d5279da4973ed17a0247733e660"}
Jul 16 22:37:56.941871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:56.941798    2562 scope.go:117] "RemoveContainer" containerID="9045c19c40e2ddf86ca80693ac062205ae40e26177d4495623c992c076b1c140"
Jul 16 22:37:57.949533 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:57.949500    2562 generic.go:356] "Generic (PLEG): container finished" podID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerID="0e922c4701898f6c9dbd0f510d3f542993b721ccac332de5b9adf4f7ef6fbae0" exitCode=6
Jul 16 22:37:57.950021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:57.949613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" event={"ID":"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7","Type":"ContainerDied","Data":"0e922c4701898f6c9dbd0f510d3f542993b721ccac332de5b9adf4f7ef6fbae0"}
Jul 16 22:37:57.950021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:57.949704    2562 scope.go:117] "RemoveContainer" containerID="9045c19c40e2ddf86ca80693ac062205ae40e26177d4495623c992c076b1c140"
Jul 16 22:37:57.950021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:57.949940    2562 scope.go:117] "RemoveContainer" containerID="0e922c4701898f6c9dbd0f510d3f542993b721ccac332de5b9adf4f7ef6fbae0"
Jul 16 22:37:57.950201 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:37:57.950177    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx_odh-ai-gateway-infra(98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7"
Jul 16 22:37:58.955817 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:37:58.955790    2562 scope.go:117] "RemoveContainer" containerID="0e922c4701898f6c9dbd0f510d3f542993b721ccac332de5b9adf4f7ef6fbae0"
Jul 16 22:37:58.956177 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:37:58.955970    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx_odh-ai-gateway-infra(98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7"
Jul 16 22:38:05.604607 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:05.604573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:05.707217 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:05.707180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:13.745137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:13.745099    2562 scope.go:117] "RemoveContainer" containerID="0e922c4701898f6c9dbd0f510d3f542993b721ccac332de5b9adf4f7ef6fbae0"
Jul 16 22:38:14.025701 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:14.025615    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" event={"ID":"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7","Type":"ContainerStarted","Data":"fc4c7b812939e0f89390e9a104c8f741488128159537f10d8d485bef3f5b9c82"}
Jul 16 22:38:14.047347 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:14.047290    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" podStartSLOduration=19.047271146 podStartE2EDuration="19.047271146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:14.045688311 +0000 UTC m=+3076.827586159" watchObservedRunningTime="2026-07-16 22:38:14.047271146 +0000 UTC m=+3076.829169008"
Jul 16 22:38:15.032337 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.032307    2562 generic.go:356] "Generic (PLEG): container finished" podID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerID="fc4c7b812939e0f89390e9a104c8f741488128159537f10d8d485bef3f5b9c82" exitCode=6
Jul 16 22:38:15.032337 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.032345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" event={"ID":"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7","Type":"ContainerDied","Data":"fc4c7b812939e0f89390e9a104c8f741488128159537f10d8d485bef3f5b9c82"}
Jul 16 22:38:15.032905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.032404    2562 scope.go:117] "RemoveContainer" containerID="0e922c4701898f6c9dbd0f510d3f542993b721ccac332de5b9adf4f7ef6fbae0"
Jul 16 22:38:15.032905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.032862    2562 scope.go:117] "RemoveContainer" containerID="fc4c7b812939e0f89390e9a104c8f741488128159537f10d8d485bef3f5b9c82"
Jul 16 22:38:15.033129 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:38:15.033105    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx_odh-ai-gateway-infra(98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7"
Jul 16 22:38:15.066430 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.066397    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"]
Jul 16 22:38:15.609631 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.609598    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"]
Jul 16 22:38:15.614437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.614414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.618438 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.618413    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3a3e4b\"/\"openshift-service-ca.crt\""
Jul 16 22:38:15.618438 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.618434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3a3e4b\"/\"default-dockercfg-jm8pd\""
Jul 16 22:38:15.618646 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.618434    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3a3e4b\"/\"kube-root-ca.crt\""
Jul 16 22:38:15.618646 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.618570    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3a3e4b\"/\"test-model-3a3e4b-kserve-self-signed-certs\""
Jul 16 22:38:15.627007 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.626984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"]
Jul 16 22:38:15.642492 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.642461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:15.802400 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.802360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-model-cache\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.802590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.802413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-tmp-dir\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.802590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.802483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc363d4-f715-4a58-bc22-24e2367fdba3-tls-certs\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.802590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.802529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-home\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.802590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.802567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsppr\" (UniqueName: \"kubernetes.io/projected/bbc363d4-f715-4a58-bc22-24e2367fdba3-kube-api-access-fsppr\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.802734 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.802660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-dshm\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.903905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.903880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc363d4-f715-4a58-bc22-24e2367fdba3-tls-certs\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904064 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.903941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-home\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904064 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.903966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsppr\" (UniqueName: \"kubernetes.io/projected/bbc363d4-f715-4a58-bc22-24e2367fdba3-kube-api-access-fsppr\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904064 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.904006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-dshm\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904064 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.904052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-model-cache\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.904092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-tmp-dir\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.904416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-home\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904504 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.904473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-model-cache\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.904504 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.904490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-tmp-dir\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.906314 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.906289    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-dshm\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.906435 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.906419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc363d4-f715-4a58-bc22-24e2367fdba3-tls-certs\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.912671 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.912653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsppr\" (UniqueName: \"kubernetes.io/projected/bbc363d4-f715-4a58-bc22-24e2367fdba3-kube-api-access-fsppr\") pod \"test-model-3a3e4b-kserve-7448855d77-k57fg\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") " pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:15.923628 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:15.923602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:16.062684 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:16.062656    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"]
Jul 16 22:38:16.063964 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:38:16.063933    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc363d4_f715_4a58_bc22_24e2367fdba3.slice/crio-cd2366cda8354ddca6c64469704cb47dbcd0a94fcb478212a1600fbab71dd103 WatchSource:0}: Error finding container cd2366cda8354ddca6c64469704cb47dbcd0a94fcb478212a1600fbab71dd103: Status 404 returned error can't find the container with id cd2366cda8354ddca6c64469704cb47dbcd0a94fcb478212a1600fbab71dd103
Jul 16 22:38:16.169499 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:16.169472    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:38:16.307460 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:16.307422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7-maas-api-service-ca\") pod \"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7\" (UID: \"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7\") "
Jul 16 22:38:16.307803 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:16.307774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" (UID: "98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:16.408327 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:16.408300    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:38:17.044394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.044363    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"
Jul 16 22:38:17.044609 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.044364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx" event={"ID":"98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7","Type":"ContainerDied","Data":"f89166bc59b50e577f3d086be2dfbd3fe0608d5279da4973ed17a0247733e660"}
Jul 16 22:38:17.044609 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.044516    2562 scope.go:117] "RemoveContainer" containerID="fc4c7b812939e0f89390e9a104c8f741488128159537f10d8d485bef3f5b9c82"
Jul 16 22:38:17.046268 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.046245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg" event={"ID":"bbc363d4-f715-4a58-bc22-24e2367fdba3","Type":"ContainerStarted","Data":"172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6"}
Jul 16 22:38:17.046386 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.046279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg" event={"ID":"bbc363d4-f715-4a58-bc22-24e2367fdba3","Type":"ContainerStarted","Data":"cd2366cda8354ddca6c64469704cb47dbcd0a94fcb478212a1600fbab71dd103"}
Jul 16 22:38:17.071506 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.071452    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg" podStartSLOduration=2.07143807 podStartE2EDuration="2.07143807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:17.069924935 +0000 UTC m=+3079.851822782" watchObservedRunningTime="2026-07-16 22:38:17.07143807 +0000 UTC m=+3079.853335942"
Jul 16 22:38:17.086232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.086206    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"]
Jul 16 22:38:17.092089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.092070    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gdtdx"]
Jul 16 22:38:17.751823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:17.751794    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" path="/var/lib/kubelet/pods/98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7/volumes"
Jul 16 22:38:25.924750 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:25.924702    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:25.925218 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:25.924840    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:25.937369 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:25.937348    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:26.102067 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:26.102037    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:38:42.225303 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225268    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"]
Jul 16 22:38:42.225772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225652    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:38:42.225772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225664    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:38:42.225772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225672    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:38:42.225772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225679    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:38:42.225772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225737    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:38:42.225772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.225747    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:38:42.228364 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.228346    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.233424 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.233395    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2fe960\"/\"openshift-service-ca.crt\""
Jul 16 22:38:42.233540 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.233427    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2fe960\"/\"test-model-2fe960-kserve-self-signed-certs\""
Jul 16 22:38:42.233540 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.233428    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2fe960\"/\"default-dockercfg-6t6cv\""
Jul 16 22:38:42.233680 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.233395    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2fe960\"/\"kube-root-ca.crt\""
Jul 16 22:38:42.245820 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.245794    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"]
Jul 16 22:38:42.278085 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.278045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:42.345013 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.344979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rmz\" (UniqueName: \"kubernetes.io/projected/d1bf90ec-42cd-43f3-9b40-74c316917a50-kube-api-access-l2rmz\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.345199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.345028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-dshm\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.345199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.345059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-tmp-dir\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.345199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.345110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1bf90ec-42cd-43f3-9b40-74c316917a50-tls-certs\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.345199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.345126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-model-cache\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.345199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.345144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-home\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.445827 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.445785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rmz\" (UniqueName: \"kubernetes.io/projected/d1bf90ec-42cd-43f3-9b40-74c316917a50-kube-api-access-l2rmz\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.445863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-dshm\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.445909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-tmp-dir\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.445948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1bf90ec-42cd-43f3-9b40-74c316917a50-tls-certs\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.445976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-model-cache\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.446004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-home\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446381 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.446349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-tmp-dir\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446451 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.446389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-home\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.446451 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.446410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-model-cache\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.448423 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.448402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-dshm\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.448585 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.448489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1bf90ec-42cd-43f3-9b40-74c316917a50-tls-certs\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.458889 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.458865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rmz\" (UniqueName: \"kubernetes.io/projected/d1bf90ec-42cd-43f3-9b40-74c316917a50-kube-api-access-l2rmz\") pod \"test-model-2fe960-kserve-86f647455c-7r46g\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") " pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.538069 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.537993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:42.675468 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:42.675440    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"]
Jul 16 22:38:42.677724 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:38:42.677699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1bf90ec_42cd_43f3_9b40_74c316917a50.slice/crio-01e765ec9b91465691de3c86d3c57c2418670a4e911c500ac9a0c9ae73c7b86f WatchSource:0}: Error finding container 01e765ec9b91465691de3c86d3c57c2418670a4e911c500ac9a0c9ae73c7b86f: Status 404 returned error can't find the container with id 01e765ec9b91465691de3c86d3c57c2418670a4e911c500ac9a0c9ae73c7b86f
Jul 16 22:38:43.181174 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:43.181139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g" event={"ID":"d1bf90ec-42cd-43f3-9b40-74c316917a50","Type":"ContainerStarted","Data":"fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a"}
Jul 16 22:38:43.181354 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:43.181183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g" event={"ID":"d1bf90ec-42cd-43f3-9b40-74c316917a50","Type":"ContainerStarted","Data":"01e765ec9b91465691de3c86d3c57c2418670a4e911c500ac9a0c9ae73c7b86f"}
Jul 16 22:38:52.538786 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:52.538727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:52.538786 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:52.538802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:52.551952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:52.551927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:38:52.568468 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:52.568415    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g" podStartSLOduration=10.56839959 podStartE2EDuration="10.56839959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:43.237258685 +0000 UTC m=+3106.019156533" watchObservedRunningTime="2026-07-16 22:38:52.56839959 +0000 UTC m=+3115.350297472"
Jul 16 22:38:53.241610 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:38:53.241584    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:39:27.541498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:27.541458    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"]
Jul 16 22:39:27.565336 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:27.565282    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:28.464540 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:28.464509    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"]
Jul 16 22:39:28.488703 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:28.488662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:29.323980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:29.323950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:29.405429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:29.405382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:39.193955 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.193919    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"]
Jul 16 22:39:39.194421 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.194135    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" podUID="860a3dbc-d2e0-4163-b010-786c96122bf8" containerName="maas-api" containerID="cri-o://4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e" gracePeriod=30
Jul 16 22:39:39.239138 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.239107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:39.631853 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.631825    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:39:39.777642 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.777541    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls\") pod \"860a3dbc-d2e0-4163-b010-786c96122bf8\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") "
Jul 16 22:39:39.777642 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.777607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860a3dbc-d2e0-4163-b010-786c96122bf8-openshift-service-ca\") pod \"860a3dbc-d2e0-4163-b010-786c96122bf8\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") "
Jul 16 22:39:39.777859 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.777670    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jpjz\" (UniqueName: \"kubernetes.io/projected/860a3dbc-d2e0-4163-b010-786c96122bf8-kube-api-access-6jpjz\") pod \"860a3dbc-d2e0-4163-b010-786c96122bf8\" (UID: \"860a3dbc-d2e0-4163-b010-786c96122bf8\") "
Jul 16 22:39:39.777974 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.777950    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860a3dbc-d2e0-4163-b010-786c96122bf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "860a3dbc-d2e0-4163-b010-786c96122bf8" (UID: "860a3dbc-d2e0-4163-b010-786c96122bf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:39.779757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.779736    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "860a3dbc-d2e0-4163-b010-786c96122bf8" (UID: "860a3dbc-d2e0-4163-b010-786c96122bf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:39:39.779902 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.779882    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860a3dbc-d2e0-4163-b010-786c96122bf8-kube-api-access-6jpjz" (OuterVolumeSpecName: "kube-api-access-6jpjz") pod "860a3dbc-d2e0-4163-b010-786c96122bf8" (UID: "860a3dbc-d2e0-4163-b010-786c96122bf8"). InnerVolumeSpecName "kube-api-access-6jpjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:39.878499 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.878467    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860a3dbc-d2e0-4163-b010-786c96122bf8-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:39.878499 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.878495    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860a3dbc-d2e0-4163-b010-786c96122bf8-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:39.878749 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:39.878509    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jpjz\" (UniqueName: \"kubernetes.io/projected/860a3dbc-d2e0-4163-b010-786c96122bf8-kube-api-access-6jpjz\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:40.453699 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.453663    2562 generic.go:356] "Generic (PLEG): container finished" podID="860a3dbc-d2e0-4163-b010-786c96122bf8" containerID="4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e" exitCode=0
Jul 16 22:39:40.454182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.453722    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"
Jul 16 22:39:40.454182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.453728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" event={"ID":"860a3dbc-d2e0-4163-b010-786c96122bf8","Type":"ContainerDied","Data":"4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e"}
Jul 16 22:39:40.454182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.453844    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f" event={"ID":"860a3dbc-d2e0-4163-b010-786c96122bf8","Type":"ContainerDied","Data":"577f99b44267e518eb68ac3aca8639e7dbae26016effb569c85a4c5d349b9c99"}
Jul 16 22:39:40.454182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.453867    2562 scope.go:117] "RemoveContainer" containerID="4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e"
Jul 16 22:39:40.463820 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.463802    2562 scope.go:117] "RemoveContainer" containerID="4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e"
Jul 16 22:39:40.464104 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:39:40.464086    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e\": container with ID starting with 4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e not found: ID does not exist" containerID="4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e"
Jul 16 22:39:40.464163 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.464112    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e"} err="failed to get container status \"4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e\": rpc error: code = NotFound desc = could not find container \"4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e\": container with ID starting with 4cc61e9a1c07bde8fbeb03f68cf767ec0cf87f542f744c97169ba8638062f29e not found: ID does not exist"
Jul 16 22:39:40.476475 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.476449    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"]
Jul 16 22:39:40.482039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:40.482014    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a3e4b-8bf796789-tlf5f"]
Jul 16 22:39:41.752970 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:41.752937    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860a3dbc-d2e0-4163-b010-786c96122bf8" path="/var/lib/kubelet/pods/860a3dbc-d2e0-4163-b010-786c96122bf8/volumes"
Jul 16 22:39:42.562586 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.562506    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg" podUID="bbc363d4-f715-4a58-bc22-24e2367fdba3" containerName="main" containerID="cri-o://172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6" gracePeriod=45
Jul 16 22:39:42.815919 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.815853    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:39:42.904355 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904321    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-model-cache\") pod \"bbc363d4-f715-4a58-bc22-24e2367fdba3\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") "
Jul 16 22:39:42.904355 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904353    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-dshm\") pod \"bbc363d4-f715-4a58-bc22-24e2367fdba3\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") "
Jul 16 22:39:42.904638 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904378    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-home\") pod \"bbc363d4-f715-4a58-bc22-24e2367fdba3\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") "
Jul 16 22:39:42.904638 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904401    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-tmp-dir\") pod \"bbc363d4-f715-4a58-bc22-24e2367fdba3\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") "
Jul 16 22:39:42.904638 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904474    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc363d4-f715-4a58-bc22-24e2367fdba3-tls-certs\") pod \"bbc363d4-f715-4a58-bc22-24e2367fdba3\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") "
Jul 16 22:39:42.904638 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904505    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsppr\" (UniqueName: \"kubernetes.io/projected/bbc363d4-f715-4a58-bc22-24e2367fdba3-kube-api-access-fsppr\") pod \"bbc363d4-f715-4a58-bc22-24e2367fdba3\" (UID: \"bbc363d4-f715-4a58-bc22-24e2367fdba3\") "
Jul 16 22:39:42.904847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904673    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbc363d4-f715-4a58-bc22-24e2367fdba3" (UID: "bbc363d4-f715-4a58-bc22-24e2367fdba3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:42.904847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904685    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-home" (OuterVolumeSpecName: "home") pod "bbc363d4-f715-4a58-bc22-24e2367fdba3" (UID: "bbc363d4-f715-4a58-bc22-24e2367fdba3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:42.904847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904730    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbc363d4-f715-4a58-bc22-24e2367fdba3" (UID: "bbc363d4-f715-4a58-bc22-24e2367fdba3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:42.904847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904777    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-model-cache\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:42.904847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.904788    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-home\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:42.906889 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.906861    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-dshm" (OuterVolumeSpecName: "dshm") pod "bbc363d4-f715-4a58-bc22-24e2367fdba3" (UID: "bbc363d4-f715-4a58-bc22-24e2367fdba3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:42.906985 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.906888    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc363d4-f715-4a58-bc22-24e2367fdba3-kube-api-access-fsppr" (OuterVolumeSpecName: "kube-api-access-fsppr") pod "bbc363d4-f715-4a58-bc22-24e2367fdba3" (UID: "bbc363d4-f715-4a58-bc22-24e2367fdba3"). InnerVolumeSpecName "kube-api-access-fsppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:42.906985 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:42.906917    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc363d4-f715-4a58-bc22-24e2367fdba3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbc363d4-f715-4a58-bc22-24e2367fdba3" (UID: "bbc363d4-f715-4a58-bc22-24e2367fdba3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:39:43.011153 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.005898    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc363d4-f715-4a58-bc22-24e2367fdba3-tls-certs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.011153 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.005938    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsppr\" (UniqueName: \"kubernetes.io/projected/bbc363d4-f715-4a58-bc22-24e2367fdba3-kube-api-access-fsppr\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.011153 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.005955    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-dshm\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.011153 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.005968    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc363d4-f715-4a58-bc22-24e2367fdba3-tmp-dir\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.471196 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.471164    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbc363d4-f715-4a58-bc22-24e2367fdba3" containerID="172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6" exitCode=0
Jul 16 22:39:43.471408 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.471253    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"
Jul 16 22:39:43.471408 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.471254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg" event={"ID":"bbc363d4-f715-4a58-bc22-24e2367fdba3","Type":"ContainerDied","Data":"172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6"}
Jul 16 22:39:43.471408 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.471307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg" event={"ID":"bbc363d4-f715-4a58-bc22-24e2367fdba3","Type":"ContainerDied","Data":"cd2366cda8354ddca6c64469704cb47dbcd0a94fcb478212a1600fbab71dd103"}
Jul 16 22:39:43.471408 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.471327    2562 scope.go:117] "RemoveContainer" containerID="172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6"
Jul 16 22:39:43.480807 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.480790    2562 scope.go:117] "RemoveContainer" containerID="172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6"
Jul 16 22:39:43.481079 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:39:43.481058    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6\": container with ID starting with 172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6 not found: ID does not exist" containerID="172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6"
Jul 16 22:39:43.481124 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.481086    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6"} err="failed to get container status \"172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6\": rpc error: code = NotFound desc = could not find container \"172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6\": container with ID starting with 172623c757feb4dbe8baf0ab6ae3295a1ca805c57c37cdfbdcac368dc8ddfbd6 not found: ID does not exist"
Jul 16 22:39:43.484822 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.484797    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g" podUID="d1bf90ec-42cd-43f3-9b40-74c316917a50" containerName="main" containerID="cri-o://fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a" gracePeriod=45
Jul 16 22:39:43.495128 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.495106    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"]
Jul 16 22:39:43.497061 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.497041    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3a3e4b/test-model-3a3e4b-kserve-7448855d77-k57fg"]
Jul 16 22:39:43.734113 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.734039    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:39:43.734972 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.734945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:43.752489 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.752453    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc363d4-f715-4a58-bc22-24e2367fdba3" path="/var/lib/kubelet/pods/bbc363d4-f715-4a58-bc22-24e2367fdba3/volumes"
Jul 16 22:39:43.813190 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-dshm\") pod \"d1bf90ec-42cd-43f3-9b40-74c316917a50\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") "
Jul 16 22:39:43.813190 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1bf90ec-42cd-43f3-9b40-74c316917a50-tls-certs\") pod \"d1bf90ec-42cd-43f3-9b40-74c316917a50\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") "
Jul 16 22:39:43.813429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813243    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-model-cache\") pod \"d1bf90ec-42cd-43f3-9b40-74c316917a50\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") "
Jul 16 22:39:43.813429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813271    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-home\") pod \"d1bf90ec-42cd-43f3-9b40-74c316917a50\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") "
Jul 16 22:39:43.813429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-tmp-dir\") pod \"d1bf90ec-42cd-43f3-9b40-74c316917a50\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") "
Jul 16 22:39:43.813429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813374    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2rmz\" (UniqueName: \"kubernetes.io/projected/d1bf90ec-42cd-43f3-9b40-74c316917a50-kube-api-access-l2rmz\") pod \"d1bf90ec-42cd-43f3-9b40-74c316917a50\" (UID: \"d1bf90ec-42cd-43f3-9b40-74c316917a50\") "
Jul 16 22:39:43.813689 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-home" (OuterVolumeSpecName: "home") pod "d1bf90ec-42cd-43f3-9b40-74c316917a50" (UID: "d1bf90ec-42cd-43f3-9b40-74c316917a50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:43.813689 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1bf90ec-42cd-43f3-9b40-74c316917a50" (UID: "d1bf90ec-42cd-43f3-9b40-74c316917a50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:43.813784 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1bf90ec-42cd-43f3-9b40-74c316917a50" (UID: "d1bf90ec-42cd-43f3-9b40-74c316917a50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:43.813784 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813717    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-model-cache\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.813784 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.813735    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-home\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.815713 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.815681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1bf90ec-42cd-43f3-9b40-74c316917a50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1bf90ec-42cd-43f3-9b40-74c316917a50" (UID: "d1bf90ec-42cd-43f3-9b40-74c316917a50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:39:43.815713 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.815677    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-dshm" (OuterVolumeSpecName: "dshm") pod "d1bf90ec-42cd-43f3-9b40-74c316917a50" (UID: "d1bf90ec-42cd-43f3-9b40-74c316917a50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:43.815861 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.815797    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bf90ec-42cd-43f3-9b40-74c316917a50-kube-api-access-l2rmz" (OuterVolumeSpecName: "kube-api-access-l2rmz") pod "d1bf90ec-42cd-43f3-9b40-74c316917a50" (UID: "d1bf90ec-42cd-43f3-9b40-74c316917a50"). InnerVolumeSpecName "kube-api-access-l2rmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:43.915021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.914987    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2rmz\" (UniqueName: \"kubernetes.io/projected/d1bf90ec-42cd-43f3-9b40-74c316917a50-kube-api-access-l2rmz\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.915021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.915017    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-dshm\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.915021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.915027    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1bf90ec-42cd-43f3-9b40-74c316917a50-tls-certs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:43.915457 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:43.915059    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1bf90ec-42cd-43f3-9b40-74c316917a50-tmp-dir\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:39:44.477137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.477101    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1bf90ec-42cd-43f3-9b40-74c316917a50" containerID="fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a" exitCode=0
Jul 16 22:39:44.477137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.477139    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"
Jul 16 22:39:44.477418 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.477184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g" event={"ID":"d1bf90ec-42cd-43f3-9b40-74c316917a50","Type":"ContainerDied","Data":"fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a"}
Jul 16 22:39:44.477418 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.477223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g" event={"ID":"d1bf90ec-42cd-43f3-9b40-74c316917a50","Type":"ContainerDied","Data":"01e765ec9b91465691de3c86d3c57c2418670a4e911c500ac9a0c9ae73c7b86f"}
Jul 16 22:39:44.477418 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.477244    2562 scope.go:117] "RemoveContainer" containerID="fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a"
Jul 16 22:39:44.488252 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.488231    2562 scope.go:117] "RemoveContainer" containerID="fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a"
Jul 16 22:39:44.488508 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:39:44.488489    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a\": container with ID starting with fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a not found: ID does not exist" containerID="fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a"
Jul 16 22:39:44.488583 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.488518    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a"} err="failed to get container status \"fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a\": rpc error: code = NotFound desc = could not find container \"fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a\": container with ID starting with fc1987502b1a5e239f435f4546a774a278e25ea8cea88c33ecf6f2635c23c18a not found: ID does not exist"
Jul 16 22:39:44.503942 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.503914    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"]
Jul 16 22:39:44.518596 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:44.518571    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2fe960/test-model-2fe960-kserve-86f647455c-7r46g"]
Jul 16 22:39:45.753018 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:45.752980    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1bf90ec-42cd-43f3-9b40-74c316917a50" path="/var/lib/kubelet/pods/d1bf90ec-42cd-43f3-9b40-74c316917a50/volumes"
Jul 16 22:39:50.726865 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:50.726829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:50.918269 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:50.918231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:54.443729 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:54.443693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:54.672651 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:54.672617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:55.313243 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.313199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:55.408700 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.408631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:55.878092 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878064    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"]
Jul 16 22:39:55.878639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878572    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc363d4-f715-4a58-bc22-24e2367fdba3" containerName="main"
Jul 16 22:39:55.878639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878595    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc363d4-f715-4a58-bc22-24e2367fdba3" containerName="main"
Jul 16 22:39:55.878639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878607    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bf90ec-42cd-43f3-9b40-74c316917a50" containerName="main"
Jul 16 22:39:55.878639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878615    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bf90ec-42cd-43f3-9b40-74c316917a50" containerName="main"
Jul 16 22:39:55.878639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878641    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860a3dbc-d2e0-4163-b010-786c96122bf8" containerName="maas-api"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878649    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="860a3dbc-d2e0-4163-b010-786c96122bf8" containerName="maas-api"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878668    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878677    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878755    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e3ef20-b1b0-4ee3-a99c-7b2d0ac160c7" containerName="revoke-keys"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878769    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bf90ec-42cd-43f3-9b40-74c316917a50" containerName="main"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878782    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="860a3dbc-d2e0-4163-b010-786c96122bf8" containerName="maas-api"
Jul 16 22:39:55.878918 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.878791    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc363d4-f715-4a58-bc22-24e2367fdba3" containerName="main"
Jul 16 22:39:55.883285 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.883265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:39:55.885936 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.885919    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:39:55.890120 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.890096    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"]
Jul 16 22:39:55.926439 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:55.926403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be69710d-6a85-41d6-9ca7-ca988d283bbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw\" (UID: \"be69710d-6a85-41d6-9ca7-ca988d283bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:39:56.027486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:56.027450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be69710d-6a85-41d6-9ca7-ca988d283bbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw\" (UID: \"be69710d-6a85-41d6-9ca7-ca988d283bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:39:56.028095 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:56.028072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be69710d-6a85-41d6-9ca7-ca988d283bbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw\" (UID: \"be69710d-6a85-41d6-9ca7-ca988d283bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:39:56.195296 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:56.195263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:39:56.328179 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:56.328138    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"]
Jul 16 22:39:56.330524 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:39:56.330499    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe69710d_6a85_41d6_9ca7_ca988d283bbf.slice/crio-2e58976f94ed6dc00027029116cb2f020fa102524bd5e03a4392d56c6e01e2b5 WatchSource:0}: Error finding container 2e58976f94ed6dc00027029116cb2f020fa102524bd5e03a4392d56c6e01e2b5: Status 404 returned error can't find the container with id 2e58976f94ed6dc00027029116cb2f020fa102524bd5e03a4392d56c6e01e2b5
Jul 16 22:39:56.537529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:56.537442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" event={"ID":"be69710d-6a85-41d6-9ca7-ca988d283bbf","Type":"ContainerStarted","Data":"2e58976f94ed6dc00027029116cb2f020fa102524bd5e03a4392d56c6e01e2b5"}
Jul 16 22:39:57.543996 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:57.543956    2562 generic.go:356] "Generic (PLEG): container finished" podID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerID="27c0893b2ef4e4bb1e8f3d3a47183c3179befdb183fe81c7c53458c6c8817c6b" exitCode=6
Jul 16 22:39:57.544420 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:57.544027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" event={"ID":"be69710d-6a85-41d6-9ca7-ca988d283bbf","Type":"ContainerDied","Data":"27c0893b2ef4e4bb1e8f3d3a47183c3179befdb183fe81c7c53458c6c8817c6b"}
Jul 16 22:39:57.544420 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:57.544274    2562 scope.go:117] "RemoveContainer" containerID="27c0893b2ef4e4bb1e8f3d3a47183c3179befdb183fe81c7c53458c6c8817c6b"
Jul 16 22:39:58.551123 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:58.551086    2562 generic.go:356] "Generic (PLEG): container finished" podID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerID="f23a61c5828424078336946288b4d51959e188c2fee5447988fa226a04a53314" exitCode=6
Jul 16 22:39:58.551759 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:58.551172    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" event={"ID":"be69710d-6a85-41d6-9ca7-ca988d283bbf","Type":"ContainerDied","Data":"f23a61c5828424078336946288b4d51959e188c2fee5447988fa226a04a53314"}
Jul 16 22:39:58.551759 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:58.551239    2562 scope.go:117] "RemoveContainer" containerID="27c0893b2ef4e4bb1e8f3d3a47183c3179befdb183fe81c7c53458c6c8817c6b"
Jul 16 22:39:58.551759 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:58.551461    2562 scope.go:117] "RemoveContainer" containerID="f23a61c5828424078336946288b4d51959e188c2fee5447988fa226a04a53314"
Jul 16 22:39:58.551759 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:39:58.551668    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw_odh-ai-gateway-infra(be69710d-6a85-41d6-9ca7-ca988d283bbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf"
Jul 16 22:39:59.557145 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:39:59.557113    2562 scope.go:117] "RemoveContainer" containerID="f23a61c5828424078336946288b4d51959e188c2fee5447988fa226a04a53314"
Jul 16 22:39:59.557584 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:39:59.557307    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw_odh-ai-gateway-infra(be69710d-6a85-41d6-9ca7-ca988d283bbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf"
Jul 16 22:40:05.431224 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.431189    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"]
Jul 16 22:40:05.431722 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.431435    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" podUID="4560a29b-2e14-4ee2-a150-75d7e7624a9c" containerName="maas-api" containerID="cri-o://84ded6249a3fbcda62f4bb154a105a59d0ed1d58643b2d60ef29215dc515b166" gracePeriod=30
Jul 16 22:40:05.463921 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.463812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:05.586752 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.586718    2562 generic.go:356] "Generic (PLEG): container finished" podID="4560a29b-2e14-4ee2-a150-75d7e7624a9c" containerID="84ded6249a3fbcda62f4bb154a105a59d0ed1d58643b2d60ef29215dc515b166" exitCode=0
Jul 16 22:40:05.586912 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.586789    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" event={"ID":"4560a29b-2e14-4ee2-a150-75d7e7624a9c","Type":"ContainerDied","Data":"84ded6249a3fbcda62f4bb154a105a59d0ed1d58643b2d60ef29215dc515b166"}
Jul 16 22:40:05.681468 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.681406    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:40:05.814793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.814758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560a29b-2e14-4ee2-a150-75d7e7624a9c-openshift-service-ca\") pod \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") "
Jul 16 22:40:05.814956 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.814847    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls\") pod \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") "
Jul 16 22:40:05.814956 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.814878    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwc89\" (UniqueName: \"kubernetes.io/projected/4560a29b-2e14-4ee2-a150-75d7e7624a9c-kube-api-access-bwc89\") pod \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\" (UID: \"4560a29b-2e14-4ee2-a150-75d7e7624a9c\") "
Jul 16 22:40:05.815203 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.815174    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4560a29b-2e14-4ee2-a150-75d7e7624a9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4560a29b-2e14-4ee2-a150-75d7e7624a9c" (UID: "4560a29b-2e14-4ee2-a150-75d7e7624a9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:05.817061 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.817038    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4560a29b-2e14-4ee2-a150-75d7e7624a9c" (UID: "4560a29b-2e14-4ee2-a150-75d7e7624a9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:05.817061 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.817035    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4560a29b-2e14-4ee2-a150-75d7e7624a9c-kube-api-access-bwc89" (OuterVolumeSpecName: "kube-api-access-bwc89") pod "4560a29b-2e14-4ee2-a150-75d7e7624a9c" (UID: "4560a29b-2e14-4ee2-a150-75d7e7624a9c"). InnerVolumeSpecName "kube-api-access-bwc89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:05.916538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.916505    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4560a29b-2e14-4ee2-a150-75d7e7624a9c-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:05.916538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.916535    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwc89\" (UniqueName: \"kubernetes.io/projected/4560a29b-2e14-4ee2-a150-75d7e7624a9c-kube-api-access-bwc89\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:05.916538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:05.916560    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4560a29b-2e14-4ee2-a150-75d7e7624a9c-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:06.308177 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.308118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:06.466987 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.466950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:06.592976 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.592893    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"
Jul 16 22:40:06.592976 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.592904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm" event={"ID":"4560a29b-2e14-4ee2-a150-75d7e7624a9c","Type":"ContainerDied","Data":"a4277f2eeb1859c194d482e816795f63480a3acaa9ee6fea9504643178fbf8ee"}
Jul 16 22:40:06.593168 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.592985    2562 scope.go:117] "RemoveContainer" containerID="84ded6249a3fbcda62f4bb154a105a59d0ed1d58643b2d60ef29215dc515b166"
Jul 16 22:40:06.615506 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.615478    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"]
Jul 16 22:40:06.620099 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:06.620078    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2fe960-579c9f6cb-9g2cm"]
Jul 16 22:40:07.750376 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:07.750342    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4560a29b-2e14-4ee2-a150-75d7e7624a9c" path="/var/lib/kubelet/pods/4560a29b-2e14-4ee2-a150-75d7e7624a9c/volumes"
Jul 16 22:40:09.929718 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:09.929668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:12.745452 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:12.745411    2562 scope.go:117] "RemoveContainer" containerID="f23a61c5828424078336946288b4d51959e188c2fee5447988fa226a04a53314"
Jul 16 22:40:13.633229 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:13.633143    2562 generic.go:356] "Generic (PLEG): container finished" podID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerID="fd42f2fbe8fd79f3996fe499e9fa99b496387454ba4f02752e313d69a1ddb960" exitCode=6
Jul 16 22:40:13.633229 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:13.633201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" event={"ID":"be69710d-6a85-41d6-9ca7-ca988d283bbf","Type":"ContainerDied","Data":"fd42f2fbe8fd79f3996fe499e9fa99b496387454ba4f02752e313d69a1ddb960"}
Jul 16 22:40:13.633453 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:13.633248    2562 scope.go:117] "RemoveContainer" containerID="f23a61c5828424078336946288b4d51959e188c2fee5447988fa226a04a53314"
Jul 16 22:40:13.633573 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:13.633529    2562 scope.go:117] "RemoveContainer" containerID="fd42f2fbe8fd79f3996fe499e9fa99b496387454ba4f02752e313d69a1ddb960"
Jul 16 22:40:13.633789 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:13.633769    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw_odh-ai-gateway-infra(be69710d-6a85-41d6-9ca7-ca988d283bbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf"
Jul 16 22:40:14.671159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:14.671123    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"]
Jul 16 22:40:14.802987 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:14.802965    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:40:14.897365 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:14.897335    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be69710d-6a85-41d6-9ca7-ca988d283bbf-maas-api-service-ca\") pod \"be69710d-6a85-41d6-9ca7-ca988d283bbf\" (UID: \"be69710d-6a85-41d6-9ca7-ca988d283bbf\") "
Jul 16 22:40:14.897845 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:14.897784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be69710d-6a85-41d6-9ca7-ca988d283bbf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be69710d-6a85-41d6-9ca7-ca988d283bbf" (UID: "be69710d-6a85-41d6-9ca7-ca988d283bbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:14.998954 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:14.998880    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be69710d-6a85-41d6-9ca7-ca988d283bbf-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:15.645166 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:15.645129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw" event={"ID":"be69710d-6a85-41d6-9ca7-ca988d283bbf","Type":"ContainerDied","Data":"2e58976f94ed6dc00027029116cb2f020fa102524bd5e03a4392d56c6e01e2b5"}
Jul 16 22:40:15.645357 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:15.645207    2562 scope.go:117] "RemoveContainer" containerID="fd42f2fbe8fd79f3996fe499e9fa99b496387454ba4f02752e313d69a1ddb960"
Jul 16 22:40:15.645357 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:15.645140    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"
Jul 16 22:40:15.665983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:15.665958    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"]
Jul 16 22:40:15.670639 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:15.670616    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-786rw"]
Jul 16 22:40:15.750904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:15.750873    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" path="/var/lib/kubelet/pods/be69710d-6a85-41d6-9ca7-ca988d283bbf/volumes"
Jul 16 22:40:16.424279 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:16.424253    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:16.524889 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:16.524831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:16.596344 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:16.596238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:20.576228 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:20.576197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:20.826076 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:20.826033    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.166978 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:21.166933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.512376 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:21.512281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.968407 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:21.968378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:23.503763 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.503733    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"]
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504047    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4560a29b-2e14-4ee2-a150-75d7e7624a9c" containerName="maas-api"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504058    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4560a29b-2e14-4ee2-a150-75d7e7624a9c" containerName="maas-api"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504065    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504070    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504078    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504083    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504091    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504096    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504381 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.504381 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504176    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4560a29b-2e14-4ee2-a150-75d7e7624a9c" containerName="maas-api"
Jul 16 22:40:23.504381 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.504182    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:23.506990 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.506972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:23.509321 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.509302    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:40:23.509459 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.509321    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:40:23.514272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.513972    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"]
Jul 16 22:40:23.673194 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.673156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab4a06e-fbd1-4163-9303-91dc333079a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-vlgrg\" (UID: \"4ab4a06e-fbd1-4163-9303-91dc333079a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:23.774252 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.774150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab4a06e-fbd1-4163-9303-91dc333079a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-vlgrg\" (UID: \"4ab4a06e-fbd1-4163-9303-91dc333079a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:23.774839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.774812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab4a06e-fbd1-4163-9303-91dc333079a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-vlgrg\" (UID: \"4ab4a06e-fbd1-4163-9303-91dc333079a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:23.818105 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.818075    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:23.954077 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:23.954030    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"]
Jul 16 22:40:23.960694 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:40:23.960662    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab4a06e_fbd1_4163_9303_91dc333079a4.slice/crio-099dc813896b0e134fea36d7b658a0360cc8cc0d51813e0cf95ccde72f52b7b6 WatchSource:0}: Error finding container 099dc813896b0e134fea36d7b658a0360cc8cc0d51813e0cf95ccde72f52b7b6: Status 404 returned error can't find the container with id 099dc813896b0e134fea36d7b658a0360cc8cc0d51813e0cf95ccde72f52b7b6
Jul 16 22:40:24.681154 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.681114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:24.687982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.687949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" event={"ID":"4ab4a06e-fbd1-4163-9303-91dc333079a4","Type":"ContainerStarted","Data":"ea05a6c69df2c5bd8fe67185fd19fd2b7ed6825ab896cd9ed6ca7450d3e22755"}
Jul 16 22:40:24.688159 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.688013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" event={"ID":"4ab4a06e-fbd1-4163-9303-91dc333079a4","Type":"ContainerStarted","Data":"099dc813896b0e134fea36d7b658a0360cc8cc0d51813e0cf95ccde72f52b7b6"}
Jul 16 22:40:24.716298 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.716241    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" podStartSLOduration=1.7162243369999999 podStartE2EDuration="1.716224337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:24.711767098 +0000 UTC m=+3207.493664945" watchObservedRunningTime="2026-07-16 22:40:24.716224337 +0000 UTC m=+3207.498122184"
Jul 16 22:40:24.779204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.779170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:24.911081 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.911046    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"]
Jul 16 22:40:24.911561 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.911529    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be69710d-6a85-41d6-9ca7-ca988d283bbf" containerName="revoke-keys"
Jul 16 22:40:24.914534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.914514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:24.917731 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.917703    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a0061f-serving-cert\""
Jul 16 22:40:24.918754 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.918739    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:40:24.918980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.918965    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:40:24.931910 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.931852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"]
Jul 16 22:40:24.960882 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.960851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:24.985419 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.985378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8caf936-5fec-4b6a-ac83-2dd9553474d4-maas-api-tls\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:24.985624 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.985440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xznv\" (UniqueName: \"kubernetes.io/projected/f8caf936-5fec-4b6a-ac83-2dd9553474d4-kube-api-access-2xznv\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:24.985624 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:24.985524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8caf936-5fec-4b6a-ac83-2dd9553474d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.069156 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.069122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:25.086344 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.086317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8caf936-5fec-4b6a-ac83-2dd9553474d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.086486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.086389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8caf936-5fec-4b6a-ac83-2dd9553474d4-maas-api-tls\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.086486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.086431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xznv\" (UniqueName: \"kubernetes.io/projected/f8caf936-5fec-4b6a-ac83-2dd9553474d4-kube-api-access-2xznv\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.086975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.086959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8caf936-5fec-4b6a-ac83-2dd9553474d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.089043 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.089028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8caf936-5fec-4b6a-ac83-2dd9553474d4-maas-api-tls\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.097247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.097224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xznv\" (UniqueName: \"kubernetes.io/projected/f8caf936-5fec-4b6a-ac83-2dd9553474d4-kube-api-access-2xznv\") pod \"maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.226602 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.226491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:25.381740 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.381714    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"]
Jul 16 22:40:25.383727 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:40:25.383699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8caf936_5fec_4b6a_ac83_2dd9553474d4.slice/crio-71c674f5c1927bb709f7835fa7f4b7b6862d9c8241680598ca196c55d23197c7 WatchSource:0}: Error finding container 71c674f5c1927bb709f7835fa7f4b7b6862d9c8241680598ca196c55d23197c7: Status 404 returned error can't find the container with id 71c674f5c1927bb709f7835fa7f4b7b6862d9c8241680598ca196c55d23197c7
Jul 16 22:40:25.385184 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.385164    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:40:25.465674 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.465643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:25.565027 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.564922    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:25.698828 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.698788    2562 generic.go:356] "Generic (PLEG): container finished" podID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerID="ea05a6c69df2c5bd8fe67185fd19fd2b7ed6825ab896cd9ed6ca7450d3e22755" exitCode=6
Jul 16 22:40:25.699326 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.698871    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" event={"ID":"4ab4a06e-fbd1-4163-9303-91dc333079a4","Type":"ContainerDied","Data":"ea05a6c69df2c5bd8fe67185fd19fd2b7ed6825ab896cd9ed6ca7450d3e22755"}
Jul 16 22:40:25.699326 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.699079    2562 scope.go:117] "RemoveContainer" containerID="ea05a6c69df2c5bd8fe67185fd19fd2b7ed6825ab896cd9ed6ca7450d3e22755"
Jul 16 22:40:25.700365 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:25.700337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt" event={"ID":"f8caf936-5fec-4b6a-ac83-2dd9553474d4","Type":"ContainerStarted","Data":"71c674f5c1927bb709f7835fa7f4b7b6862d9c8241680598ca196c55d23197c7"}
Jul 16 22:40:26.707352 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.707318    2562 generic.go:356] "Generic (PLEG): container finished" podID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerID="6926ee5aac375bdc4664fc718674259b214c98abf4b7ad963b5aeb95094ee603" exitCode=6
Jul 16 22:40:26.707862 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.707404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" event={"ID":"4ab4a06e-fbd1-4163-9303-91dc333079a4","Type":"ContainerDied","Data":"6926ee5aac375bdc4664fc718674259b214c98abf4b7ad963b5aeb95094ee603"}
Jul 16 22:40:26.707862 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.707472    2562 scope.go:117] "RemoveContainer" containerID="ea05a6c69df2c5bd8fe67185fd19fd2b7ed6825ab896cd9ed6ca7450d3e22755"
Jul 16 22:40:26.707862 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.707611    2562 scope.go:117] "RemoveContainer" containerID="6926ee5aac375bdc4664fc718674259b214c98abf4b7ad963b5aeb95094ee603"
Jul 16 22:40:26.707862 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:26.707829    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-vlgrg_odh-ai-gateway-infra(4ab4a06e-fbd1-4163-9303-91dc333079a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4"
Jul 16 22:40:26.708993 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.708974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt" event={"ID":"f8caf936-5fec-4b6a-ac83-2dd9553474d4","Type":"ContainerStarted","Data":"f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98"}
Jul 16 22:40:26.709143 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.709129    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:26.818247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:26.818196    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt" podStartSLOduration=2.23499845 podStartE2EDuration="2.818181546s" totalImagesPullingTime="583.183096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:26.816540948 +0000 UTC m=+3209.598438824" watchObservedRunningTime="2026-07-16 22:40:26.818181546 +0000 UTC m=+3209.600079443"
Jul 16 22:40:27.715100 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:27.715070    2562 scope.go:117] "RemoveContainer" containerID="6926ee5aac375bdc4664fc718674259b214c98abf4b7ad963b5aeb95094ee603"
Jul 16 22:40:27.715469 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:27.715269    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-vlgrg_odh-ai-gateway-infra(4ab4a06e-fbd1-4163-9303-91dc333079a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4"
Jul 16 22:40:27.820905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:27.820868    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:27.891219 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:27.891177    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:29.149763 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:29.149718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:29.611437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:29.611354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:29.974361 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:29.974331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:30.388084 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:30.387999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:32.710264 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.710231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:32.720899 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.720876    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:32.797493 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.797459    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:32.923469 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.923439    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"]
Jul 16 22:40:32.928260 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.928240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:32.930731 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.930707    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-24b876-serving-cert\""
Jul 16 22:40:32.938720 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.938697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"]
Jul 16 22:40:32.944436 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.944408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:32.944579 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.944482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c679c5-6957-46df-a8b4-805f28bbde09-openshift-service-ca\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:32.944579 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.944532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7glp\" (UniqueName: \"kubernetes.io/projected/88c679c5-6957-46df-a8b4-805f28bbde09-kube-api-access-m7glp\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:32.964736 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:32.964669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:33.045994 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.045961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.046187 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.046030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c679c5-6957-46df-a8b4-805f28bbde09-openshift-service-ca\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.046187 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.046077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7glp\" (UniqueName: \"kubernetes.io/projected/88c679c5-6957-46df-a8b4-805f28bbde09-kube-api-access-m7glp\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.046187 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:33.046130    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-serving-cert: secret "maas-api-e2e-api-b-24b876-serving-cert" not found
Jul 16 22:40:33.046358 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:33.046209    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls podName:88c679c5-6957-46df-a8b4-805f28bbde09 nodeName:}" failed. No retries permitted until 2026-07-16 22:40:33.546186889 +0000 UTC m=+3216.328084732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls") pod "maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" (UID: "88c679c5-6957-46df-a8b4-805f28bbde09") : secret "maas-api-e2e-api-b-24b876-serving-cert" not found
Jul 16 22:40:33.046709 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.046693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c679c5-6957-46df-a8b4-805f28bbde09-openshift-service-ca\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.057183 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.057153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7glp\" (UniqueName: \"kubernetes.io/projected/88c679c5-6957-46df-a8b4-805f28bbde09-kube-api-access-m7glp\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.357602 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.357497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:33.484071 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.484022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:33.550410 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.550375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.553169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.553141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls\") pod \"maas-api-e2e-api-b-24b876-5b7d998564-jxrqf\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:33.584890 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.584862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:33.839113 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:33.839079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:34.003002 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:34.002970    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"]
Jul 16 22:40:34.013800 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:40:34.013770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c679c5_6957_46df_a8b4_805f28bbde09.slice/crio-e9040c649e517969ff6b1e888e286d1f83860989dbd1aa1275f50a8be340673b WatchSource:0}: Error finding container e9040c649e517969ff6b1e888e286d1f83860989dbd1aa1275f50a8be340673b: Status 404 returned error can't find the container with id e9040c649e517969ff6b1e888e286d1f83860989dbd1aa1275f50a8be340673b
Jul 16 22:40:34.747841 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:34.747806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" event={"ID":"88c679c5-6957-46df-a8b4-805f28bbde09","Type":"ContainerStarted","Data":"cc327ccdf6f61c52b937e37adb2db0fa0f8448cba31de57bd350c6f0587f9f7e"}
Jul 16 22:40:34.748038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:34.747856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" event={"ID":"88c679c5-6957-46df-a8b4-805f28bbde09","Type":"ContainerStarted","Data":"e9040c649e517969ff6b1e888e286d1f83860989dbd1aa1275f50a8be340673b"}
Jul 16 22:40:34.748038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:34.747919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:34.770819 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:34.770765    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" podStartSLOduration=2.413443435 podStartE2EDuration="2.770748881s" totalImagesPullingTime="357.305446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:34.769174364 +0000 UTC m=+3217.551072214" watchObservedRunningTime="2026-07-16 22:40:34.770748881 +0000 UTC m=+3217.552646761"
Jul 16 22:40:37.208431 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:37.208396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:40.745180 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:40.745148    2562 scope.go:117] "RemoveContainer" containerID="6926ee5aac375bdc4664fc718674259b214c98abf4b7ad963b5aeb95094ee603"
Jul 16 22:40:40.759891 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:40.759869    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:40:41.784359 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:41.784275    2562 generic.go:356] "Generic (PLEG): container finished" podID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerID="c5057a25a15ab432c026fcb92635a34b44627b2a74936cc2410b233bef8cc8fd" exitCode=6
Jul 16 22:40:41.784830 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:41.784357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" event={"ID":"4ab4a06e-fbd1-4163-9303-91dc333079a4","Type":"ContainerDied","Data":"c5057a25a15ab432c026fcb92635a34b44627b2a74936cc2410b233bef8cc8fd"}
Jul 16 22:40:41.784830 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:41.784431    2562 scope.go:117] "RemoveContainer" containerID="6926ee5aac375bdc4664fc718674259b214c98abf4b7ad963b5aeb95094ee603"
Jul 16 22:40:41.784830 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:41.784806    2562 scope.go:117] "RemoveContainer" containerID="c5057a25a15ab432c026fcb92635a34b44627b2a74936cc2410b233bef8cc8fd"
Jul 16 22:40:41.785133 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:41.785089    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-vlgrg_odh-ai-gateway-infra(4ab4a06e-fbd1-4163-9303-91dc333079a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4"
Jul 16 22:40:42.809595 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:42.809541    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"]
Jul 16 22:40:42.949105 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:42.949075    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:43.038873 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.038842    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab4a06e-fbd1-4163-9303-91dc333079a4-maas-api-service-ca\") pod \"4ab4a06e-fbd1-4163-9303-91dc333079a4\" (UID: \"4ab4a06e-fbd1-4163-9303-91dc333079a4\") "
Jul 16 22:40:43.039304 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.039276    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab4a06e-fbd1-4163-9303-91dc333079a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ab4a06e-fbd1-4163-9303-91dc333079a4" (UID: "4ab4a06e-fbd1-4163-9303-91dc333079a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:43.139982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.139894    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab4a06e-fbd1-4163-9303-91dc333079a4-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:43.796232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.796206    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"
Jul 16 22:40:43.796232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.796224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg" event={"ID":"4ab4a06e-fbd1-4163-9303-91dc333079a4","Type":"ContainerDied","Data":"099dc813896b0e134fea36d7b658a0360cc8cc0d51813e0cf95ccde72f52b7b6"}
Jul 16 22:40:43.796423 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.796288    2562 scope.go:117] "RemoveContainer" containerID="c5057a25a15ab432c026fcb92635a34b44627b2a74936cc2410b233bef8cc8fd"
Jul 16 22:40:43.812623 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.812596    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"]
Jul 16 22:40:43.814277 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:43.814258    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vlgrg"]
Jul 16 22:40:45.227162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227124    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"]
Jul 16 22:40:45.227700 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227679    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:40:45.227782 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227705    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:40:45.227782 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227747    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:40:45.227782 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227756    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:40:45.227929 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227831    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:40:45.227929 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.227847    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:40:45.231068 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.231051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:45.234330 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.234306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:40:45.237515 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.237491    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"]
Jul 16 22:40:45.361724 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.361690    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e3f20-1a37-450b-ae4b-6ccdfb83dcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a0061f-67lgv\" (UID: \"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:45.463067 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.463026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e3f20-1a37-450b-ae4b-6ccdfb83dcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a0061f-67lgv\" (UID: \"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:45.463778 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.463751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e3f20-1a37-450b-ae4b-6ccdfb83dcdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a0061f-67lgv\" (UID: \"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:45.544408 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.544307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:45.680443 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.680415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"]
Jul 16 22:40:45.682428 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:40:45.682388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269e3f20_1a37_450b_ae4b_6ccdfb83dcdb.slice/crio-4381de2be1b54d74cc49fe23d972e54fdbd8463c65c831bac1454c789fcbf32e WatchSource:0}: Error finding container 4381de2be1b54d74cc49fe23d972e54fdbd8463c65c831bac1454c789fcbf32e: Status 404 returned error can't find the container with id 4381de2be1b54d74cc49fe23d972e54fdbd8463c65c831bac1454c789fcbf32e
Jul 16 22:40:45.751509 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.751465    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" path="/var/lib/kubelet/pods/4ab4a06e-fbd1-4163-9303-91dc333079a4/volumes"
Jul 16 22:40:45.770012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.769967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:45.807648 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.807534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv" event={"ID":"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb","Type":"ContainerStarted","Data":"4381de2be1b54d74cc49fe23d972e54fdbd8463c65c831bac1454c789fcbf32e"}
Jul 16 22:40:45.874375 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:45.874339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:46.813654 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:46.813616    2562 generic.go:356] "Generic (PLEG): container finished" podID="269e3f20-1a37-450b-ae4b-6ccdfb83dcdb" containerID="1386f58c5b4ebb1d551da55476a93089e981a7fbfedfb8b4adf7c7e74d078b4f" exitCode=0
Jul 16 22:40:46.814141 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:46.813706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv" event={"ID":"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb","Type":"ContainerDied","Data":"1386f58c5b4ebb1d551da55476a93089e981a7fbfedfb8b4adf7c7e74d078b4f"}
Jul 16 22:40:47.946861 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:47.946834    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:48.088312 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:48.088235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e3f20-1a37-450b-ae4b-6ccdfb83dcdb-maas-api-service-ca\") pod \"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb\" (UID: \"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb\") "
Jul 16 22:40:48.088620 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:48.088599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269e3f20-1a37-450b-ae4b-6ccdfb83dcdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "269e3f20-1a37-450b-ae4b-6ccdfb83dcdb" (UID: "269e3f20-1a37-450b-ae4b-6ccdfb83dcdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:48.189567 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:48.189518    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/269e3f20-1a37-450b-ae4b-6ccdfb83dcdb-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:48.823171 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:48.823143    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv"
Jul 16 22:40:48.823351 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:48.823183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a0061f-67lgv" event={"ID":"269e3f20-1a37-450b-ae4b-6ccdfb83dcdb","Type":"ContainerDied","Data":"4381de2be1b54d74cc49fe23d972e54fdbd8463c65c831bac1454c789fcbf32e"}
Jul 16 22:40:48.823351 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:48.823225    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4381de2be1b54d74cc49fe23d972e54fdbd8463c65c831bac1454c789fcbf32e"
Jul 16 22:40:55.082378 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:55.082346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:55.182182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:55.182146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:55.880835 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:55.880805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:55.981937 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:55.981904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:56.122835 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.122800    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"]
Jul 16 22:40:56.123310 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.123059    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt" podUID="f8caf936-5fec-4b6a-ac83-2dd9553474d4" containerName="maas-api" containerID="cri-o://f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98" gracePeriod=30
Jul 16 22:40:56.155344 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.155315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:56.367387 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.367362    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:56.461829 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.461800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8caf936-5fec-4b6a-ac83-2dd9553474d4-maas-api-tls\") pod \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") "
Jul 16 22:40:56.462032 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.461907    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8caf936-5fec-4b6a-ac83-2dd9553474d4-openshift-service-ca\") pod \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") "
Jul 16 22:40:56.462032 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.461937    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xznv\" (UniqueName: \"kubernetes.io/projected/f8caf936-5fec-4b6a-ac83-2dd9553474d4-kube-api-access-2xznv\") pod \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\" (UID: \"f8caf936-5fec-4b6a-ac83-2dd9553474d4\") "
Jul 16 22:40:56.462285 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.462264    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8caf936-5fec-4b6a-ac83-2dd9553474d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8caf936-5fec-4b6a-ac83-2dd9553474d4" (UID: "f8caf936-5fec-4b6a-ac83-2dd9553474d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:56.464124 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.464096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8caf936-5fec-4b6a-ac83-2dd9553474d4-kube-api-access-2xznv" (OuterVolumeSpecName: "kube-api-access-2xznv") pod "f8caf936-5fec-4b6a-ac83-2dd9553474d4" (UID: "f8caf936-5fec-4b6a-ac83-2dd9553474d4"). InnerVolumeSpecName "kube-api-access-2xznv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:56.464244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.464094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8caf936-5fec-4b6a-ac83-2dd9553474d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8caf936-5fec-4b6a-ac83-2dd9553474d4" (UID: "f8caf936-5fec-4b6a-ac83-2dd9553474d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:56.563285 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.563255    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8caf936-5fec-4b6a-ac83-2dd9553474d4-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:56.563285 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.563279    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xznv\" (UniqueName: \"kubernetes.io/projected/f8caf936-5fec-4b6a-ac83-2dd9553474d4-kube-api-access-2xznv\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:56.563285 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.563291    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8caf936-5fec-4b6a-ac83-2dd9553474d4-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:40:56.862404 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.862305    2562 generic.go:356] "Generic (PLEG): container finished" podID="f8caf936-5fec-4b6a-ac83-2dd9553474d4" containerID="f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98" exitCode=0
Jul 16 22:40:56.862404 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.862357    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"
Jul 16 22:40:56.862404 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.862364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt" event={"ID":"f8caf936-5fec-4b6a-ac83-2dd9553474d4","Type":"ContainerDied","Data":"f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98"}
Jul 16 22:40:56.862723 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.862443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt" event={"ID":"f8caf936-5fec-4b6a-ac83-2dd9553474d4","Type":"ContainerDied","Data":"71c674f5c1927bb709f7835fa7f4b7b6862d9c8241680598ca196c55d23197c7"}
Jul 16 22:40:56.862723 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.862472    2562 scope.go:117] "RemoveContainer" containerID="f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98"
Jul 16 22:40:56.871930 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.871909    2562 scope.go:117] "RemoveContainer" containerID="f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98"
Jul 16 22:40:56.872164 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:40:56.872147    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98\": container with ID starting with f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98 not found: ID does not exist" containerID="f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98"
Jul 16 22:40:56.872228 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.872172    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98"} err="failed to get container status \"f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98\": rpc error: code = NotFound desc = could not find container \"f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98\": container with ID starting with f7fa85efe7e9a63e02454085f4d5a89c819b8a73548c7e2f3d38d23fac68af98 not found: ID does not exist"
Jul 16 22:40:56.888406 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.888385    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"]
Jul 16 22:40:56.895407 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:56.895386    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a0061f-8c985f45d-5qrrt"]
Jul 16 22:40:57.751962 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:40:57.751923    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8caf936-5fec-4b6a-ac83-2dd9553474d4" path="/var/lib/kubelet/pods/f8caf936-5fec-4b6a-ac83-2dd9553474d4/volumes"
Jul 16 22:41:00.319871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:00.319835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:05.192568 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192512    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"]
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192911    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8caf936-5fec-4b6a-ac83-2dd9553474d4" containerName="maas-api"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192926    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8caf936-5fec-4b6a-ac83-2dd9553474d4" containerName="maas-api"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192934    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269e3f20-1a37-450b-ae4b-6ccdfb83dcdb" containerName="revoke-keys"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192939    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="269e3f20-1a37-450b-ae4b-6ccdfb83dcdb" containerName="revoke-keys"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192946    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.192951    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.193009    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8caf936-5fec-4b6a-ac83-2dd9553474d4" containerName="maas-api"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.193019    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="269e3f20-1a37-450b-ae4b-6ccdfb83dcdb" containerName="revoke-keys"
Jul 16 22:41:05.193056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.193025    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab4a06e-fbd1-4163-9303-91dc333079a4" containerName="revoke-keys"
Jul 16 22:41:05.197647 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.197630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:05.201173 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.201154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:41:05.208479 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.208456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"]
Jul 16 22:41:05.341600 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.341542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28257386-a939-4549-9e4c-42c9bfaf9848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr\" (UID: \"28257386-a939-4549-9e4c-42c9bfaf9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:05.442959 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.442874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28257386-a939-4549-9e4c-42c9bfaf9848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr\" (UID: \"28257386-a939-4549-9e4c-42c9bfaf9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:05.443516 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.443494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28257386-a939-4549-9e4c-42c9bfaf9848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr\" (UID: \"28257386-a939-4549-9e4c-42c9bfaf9848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:05.508615 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.508581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:05.645901 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.645865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"]
Jul 16 22:41:05.647952 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:41:05.647927    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28257386_a939_4549_9e4c_42c9bfaf9848.slice/crio-4f78bf91e8f7d1c8ac6e9a14336e9fe23f4499a827ff483e14319b3ed47b9913 WatchSource:0}: Error finding container 4f78bf91e8f7d1c8ac6e9a14336e9fe23f4499a827ff483e14319b3ed47b9913: Status 404 returned error can't find the container with id 4f78bf91e8f7d1c8ac6e9a14336e9fe23f4499a827ff483e14319b3ed47b9913
Jul 16 22:41:05.904844 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:05.904810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" event={"ID":"28257386-a939-4549-9e4c-42c9bfaf9848","Type":"ContainerStarted","Data":"4f78bf91e8f7d1c8ac6e9a14336e9fe23f4499a827ff483e14319b3ed47b9913"}
Jul 16 22:41:06.480112 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:06.480081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:06.581696 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:06.581661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:06.911014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:06.910984    2562 generic.go:356] "Generic (PLEG): container finished" podID="28257386-a939-4549-9e4c-42c9bfaf9848" containerID="7ff2d7c5601e137b221343e65610f3e2000a35fc22229fa97a8f77293192c3d2" exitCode=6
Jul 16 22:41:06.911213 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:06.911075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" event={"ID":"28257386-a939-4549-9e4c-42c9bfaf9848","Type":"ContainerDied","Data":"7ff2d7c5601e137b221343e65610f3e2000a35fc22229fa97a8f77293192c3d2"}
Jul 16 22:41:06.911301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:06.911286    2562 scope.go:117] "RemoveContainer" containerID="7ff2d7c5601e137b221343e65610f3e2000a35fc22229fa97a8f77293192c3d2"
Jul 16 22:41:07.917641 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:07.917603    2562 generic.go:356] "Generic (PLEG): container finished" podID="28257386-a939-4549-9e4c-42c9bfaf9848" containerID="b3753b0712dbe8c8a4a75f3670866f3432bbf7ad852324033b9f6edbb59076ea" exitCode=6
Jul 16 22:41:07.918081 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:07.917692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" event={"ID":"28257386-a939-4549-9e4c-42c9bfaf9848","Type":"ContainerDied","Data":"b3753b0712dbe8c8a4a75f3670866f3432bbf7ad852324033b9f6edbb59076ea"}
Jul 16 22:41:07.918081 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:07.917745    2562 scope.go:117] "RemoveContainer" containerID="7ff2d7c5601e137b221343e65610f3e2000a35fc22229fa97a8f77293192c3d2"
Jul 16 22:41:07.918081 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:07.917846    2562 scope.go:117] "RemoveContainer" containerID="b3753b0712dbe8c8a4a75f3670866f3432bbf7ad852324033b9f6edbb59076ea"
Jul 16 22:41:07.918081 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:07.918066    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr_odh-ai-gateway-infra(28257386-a939-4549-9e4c-42c9bfaf9848)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" podUID="28257386-a939-4549-9e4c-42c9bfaf9848"
Jul 16 22:41:08.923430 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:08.923398    2562 scope.go:117] "RemoveContainer" containerID="b3753b0712dbe8c8a4a75f3670866f3432bbf7ad852324033b9f6edbb59076ea"
Jul 16 22:41:08.923925 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:08.923663    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr_odh-ai-gateway-infra(28257386-a939-4549-9e4c-42c9bfaf9848)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" podUID="28257386-a939-4549-9e4c-42c9bfaf9848"
Jul 16 22:41:11.029763 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:11.029728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:11.258445 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:11.258407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:12.079502 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:12.079469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:12.184281 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:12.184222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:21.745649 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:21.745607    2562 scope.go:117] "RemoveContainer" containerID="b3753b0712dbe8c8a4a75f3670866f3432bbf7ad852324033b9f6edbb59076ea"
Jul 16 22:41:21.831347 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:21.831314    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"]
Jul 16 22:41:21.831798 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:21.831595    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" podUID="88c679c5-6957-46df-a8b4-805f28bbde09" containerName="maas-api" containerID="cri-o://cc327ccdf6f61c52b937e37adb2db0fa0f8448cba31de57bd350c6f0587f9f7e" gracePeriod=30
Jul 16 22:41:21.859946 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:21.859917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:21.984225 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:21.984189    2562 generic.go:356] "Generic (PLEG): container finished" podID="88c679c5-6957-46df-a8b4-805f28bbde09" containerID="cc327ccdf6f61c52b937e37adb2db0fa0f8448cba31de57bd350c6f0587f9f7e" exitCode=0
Jul 16 22:41:21.984421 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:21.984261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" event={"ID":"88c679c5-6957-46df-a8b4-805f28bbde09","Type":"ContainerDied","Data":"cc327ccdf6f61c52b937e37adb2db0fa0f8448cba31de57bd350c6f0587f9f7e"}
Jul 16 22:41:22.079111 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.079088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:41:22.183364 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.183345    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls\") pod \"88c679c5-6957-46df-a8b4-805f28bbde09\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") "
Jul 16 22:41:22.183488 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.183430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c679c5-6957-46df-a8b4-805f28bbde09-openshift-service-ca\") pod \"88c679c5-6957-46df-a8b4-805f28bbde09\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") "
Jul 16 22:41:22.183488 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.183463    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7glp\" (UniqueName: \"kubernetes.io/projected/88c679c5-6957-46df-a8b4-805f28bbde09-kube-api-access-m7glp\") pod \"88c679c5-6957-46df-a8b4-805f28bbde09\" (UID: \"88c679c5-6957-46df-a8b4-805f28bbde09\") "
Jul 16 22:41:22.183789 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.183766    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c679c5-6957-46df-a8b4-805f28bbde09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88c679c5-6957-46df-a8b4-805f28bbde09" (UID: "88c679c5-6957-46df-a8b4-805f28bbde09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:22.185543 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.185517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88c679c5-6957-46df-a8b4-805f28bbde09" (UID: "88c679c5-6957-46df-a8b4-805f28bbde09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:41:22.185653 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.185581    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c679c5-6957-46df-a8b4-805f28bbde09-kube-api-access-m7glp" (OuterVolumeSpecName: "kube-api-access-m7glp") pod "88c679c5-6957-46df-a8b4-805f28bbde09" (UID: "88c679c5-6957-46df-a8b4-805f28bbde09"). InnerVolumeSpecName "kube-api-access-m7glp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:22.284162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.284087    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c679c5-6957-46df-a8b4-805f28bbde09-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:41:22.284162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.284112    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7glp\" (UniqueName: \"kubernetes.io/projected/88c679c5-6957-46df-a8b4-805f28bbde09-kube-api-access-m7glp\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:41:22.284162 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.284124    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c679c5-6957-46df-a8b4-805f28bbde09-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:41:22.989961 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.989927    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"
Jul 16 22:41:22.990424 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.989933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf" event={"ID":"88c679c5-6957-46df-a8b4-805f28bbde09","Type":"ContainerDied","Data":"e9040c649e517969ff6b1e888e286d1f83860989dbd1aa1275f50a8be340673b"}
Jul 16 22:41:22.990424 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.990045    2562 scope.go:117] "RemoveContainer" containerID="cc327ccdf6f61c52b937e37adb2db0fa0f8448cba31de57bd350c6f0587f9f7e"
Jul 16 22:41:22.999024 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.998991    2562 generic.go:356] "Generic (PLEG): container finished" podID="28257386-a939-4549-9e4c-42c9bfaf9848" containerID="f61ce40310bdc655c9c222bd13c8518c567273631e4e4e3b2e46f68a6c8fb6d8" exitCode=6
Jul 16 22:41:22.999130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.999032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" event={"ID":"28257386-a939-4549-9e4c-42c9bfaf9848","Type":"ContainerDied","Data":"f61ce40310bdc655c9c222bd13c8518c567273631e4e4e3b2e46f68a6c8fb6d8"}
Jul 16 22:41:22.999355 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:22.999340    2562 scope.go:117] "RemoveContainer" containerID="f61ce40310bdc655c9c222bd13c8518c567273631e4e4e3b2e46f68a6c8fb6d8"
Jul 16 22:41:22.999508 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:22.999492    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr_odh-ai-gateway-infra(28257386-a939-4549-9e4c-42c9bfaf9848)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" podUID="28257386-a939-4549-9e4c-42c9bfaf9848"
Jul 16 22:41:23.006082 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.006067    2562 scope.go:117] "RemoveContainer" containerID="b3753b0712dbe8c8a4a75f3670866f3432bbf7ad852324033b9f6edbb59076ea"
Jul 16 22:41:23.027791 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.027758    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"]
Jul 16 22:41:23.031315 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.031293    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24b876-5b7d998564-jxrqf"]
Jul 16 22:41:23.561803 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.561762    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"]
Jul 16 22:41:23.562180 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.562161    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c679c5-6957-46df-a8b4-805f28bbde09" containerName="maas-api"
Jul 16 22:41:23.562258 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.562183    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c679c5-6957-46df-a8b4-805f28bbde09" containerName="maas-api"
Jul 16 22:41:23.562318 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.562293    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c679c5-6957-46df-a8b4-805f28bbde09" containerName="maas-api"
Jul 16 22:41:23.566658 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.566635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:23.574058 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.574035    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"]
Jul 16 22:41:23.593780 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.593747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d79126-d679-4616-968b-a2bed6ddba7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-hh8r9\" (UID: \"d1d79126-d679-4616-968b-a2bed6ddba7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:23.695002 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.694963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d79126-d679-4616-968b-a2bed6ddba7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-hh8r9\" (UID: \"d1d79126-d679-4616-968b-a2bed6ddba7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:23.695610 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.695593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d79126-d679-4616-968b-a2bed6ddba7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-hh8r9\" (UID: \"d1d79126-d679-4616-968b-a2bed6ddba7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:23.751299 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.751265    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c679c5-6957-46df-a8b4-805f28bbde09" path="/var/lib/kubelet/pods/88c679c5-6957-46df-a8b4-805f28bbde09/volumes"
Jul 16 22:41:23.879151 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:23.879067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:24.017819 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:24.017798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"]
Jul 16 22:41:24.019248 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:41:24.019222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d79126_d679_4616_968b_a2bed6ddba7d.slice/crio-6a917b8fbf244e21262dc12931f1a8d38450e24dff1c75b98e1fe4ff59a1c477 WatchSource:0}: Error finding container 6a917b8fbf244e21262dc12931f1a8d38450e24dff1c75b98e1fe4ff59a1c477: Status 404 returned error can't find the container with id 6a917b8fbf244e21262dc12931f1a8d38450e24dff1c75b98e1fe4ff59a1c477
Jul 16 22:41:24.028322 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:24.028299    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"]
Jul 16 22:41:24.157801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:24.157780    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:24.199301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:24.199274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28257386-a939-4549-9e4c-42c9bfaf9848-maas-api-service-ca\") pod \"28257386-a939-4549-9e4c-42c9bfaf9848\" (UID: \"28257386-a939-4549-9e4c-42c9bfaf9848\") "
Jul 16 22:41:24.199747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:24.199721    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28257386-a939-4549-9e4c-42c9bfaf9848-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28257386-a939-4549-9e4c-42c9bfaf9848" (UID: "28257386-a939-4549-9e4c-42c9bfaf9848"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:24.300474 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:24.300440    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28257386-a939-4549-9e4c-42c9bfaf9848-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:41:25.012525 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.012435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr" event={"ID":"28257386-a939-4549-9e4c-42c9bfaf9848","Type":"ContainerDied","Data":"4f78bf91e8f7d1c8ac6e9a14336e9fe23f4499a827ff483e14319b3ed47b9913"}
Jul 16 22:41:25.012525 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.012501    2562 scope.go:117] "RemoveContainer" containerID="f61ce40310bdc655c9c222bd13c8518c567273631e4e4e3b2e46f68a6c8fb6d8"
Jul 16 22:41:25.012764 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.012445    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"
Jul 16 22:41:25.014118 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.014088    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerID="3b6a60315b5d754ecfeca121e4a2054c0746a6019fb140e3e3e799b046fb8f8b" exitCode=6
Jul 16 22:41:25.014271 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.014248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" event={"ID":"d1d79126-d679-4616-968b-a2bed6ddba7d","Type":"ContainerDied","Data":"3b6a60315b5d754ecfeca121e4a2054c0746a6019fb140e3e3e799b046fb8f8b"}
Jul 16 22:41:25.014342 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.014306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" event={"ID":"d1d79126-d679-4616-968b-a2bed6ddba7d","Type":"ContainerStarted","Data":"6a917b8fbf244e21262dc12931f1a8d38450e24dff1c75b98e1fe4ff59a1c477"}
Jul 16 22:41:25.014410 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.014396    2562 scope.go:117] "RemoveContainer" containerID="3b6a60315b5d754ecfeca121e4a2054c0746a6019fb140e3e3e799b046fb8f8b"
Jul 16 22:41:25.064096 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.064067    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"]
Jul 16 22:41:25.069779 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.069757    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-sx4zr"]
Jul 16 22:41:25.760720 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:25.760690    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" path="/var/lib/kubelet/pods/28257386-a939-4549-9e4c-42c9bfaf9848/volumes"
Jul 16 22:41:26.021877 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:26.021790    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerID="4433a04b93b87bf346cbc06daa76232f4d10d07638d76abe7c0221cc46b16193" exitCode=6
Jul 16 22:41:26.021877 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:26.021834    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" event={"ID":"d1d79126-d679-4616-968b-a2bed6ddba7d","Type":"ContainerDied","Data":"4433a04b93b87bf346cbc06daa76232f4d10d07638d76abe7c0221cc46b16193"}
Jul 16 22:41:26.022111 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:26.021879    2562 scope.go:117] "RemoveContainer" containerID="3b6a60315b5d754ecfeca121e4a2054c0746a6019fb140e3e3e799b046fb8f8b"
Jul 16 22:41:26.022111 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:26.022057    2562 scope.go:117] "RemoveContainer" containerID="4433a04b93b87bf346cbc06daa76232f4d10d07638d76abe7c0221cc46b16193"
Jul 16 22:41:26.022307 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:26.022284    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-hh8r9_odh-ai-gateway-infra(d1d79126-d679-4616-968b-a2bed6ddba7d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d"
Jul 16 22:41:26.470270 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:26.470242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:27.028608 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:27.028570    2562 scope.go:117] "RemoveContainer" containerID="4433a04b93b87bf346cbc06daa76232f4d10d07638d76abe7c0221cc46b16193"
Jul 16 22:41:27.028800 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:27.028779    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-hh8r9_odh-ai-gateway-infra(d1d79126-d679-4616-968b-a2bed6ddba7d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d"
Jul 16 22:41:33.031447 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:33.031409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:33.100650 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:33.100619    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:37.331202 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:37.331156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:37.555870 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:37.555838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:37.898797 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:37.898760    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:38.227029 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:38.226981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:38.396644 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:38.396610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:40.745774 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:40.745744    2562 scope.go:117] "RemoveContainer" containerID="4433a04b93b87bf346cbc06daa76232f4d10d07638d76abe7c0221cc46b16193"
Jul 16 22:41:41.095887 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.095800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" event={"ID":"d1d79126-d679-4616-968b-a2bed6ddba7d","Type":"ContainerStarted","Data":"5ea1dde8cef30612a8002165fbe4e637173919244298e1013ab9104e088e0da4"}
Jul 16 22:41:41.114430 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.114374    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" podStartSLOduration=18.114359845 podStartE2EDuration="18.114359845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:41.113028523 +0000 UTC m=+3283.894926376" watchObservedRunningTime="2026-07-16 22:41:41.114359845 +0000 UTC m=+3283.896257724"
Jul 16 22:41:41.392306 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392273    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"]
Jul 16 22:41:41.392857 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392835    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.392952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392862    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.392952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.392952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392886    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.392952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392901    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.392952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392910    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.393176 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.392994    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.393176 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.393007    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:41.396288 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.396266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.405992 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.405973    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t7zdm\""
Jul 16 22:41:41.414715 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.414693    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fbed9b-serving-cert\""
Jul 16 22:41:41.416831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.416812    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:41:41.439881 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.439861    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:41.448315 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.448247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"]
Jul 16 22:41:41.448749 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.448708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.448893 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.448808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngh8\" (UniqueName: \"kubernetes.io/projected/46da0e33-7e1d-45f2-ba29-4f9e225abccb-kube-api-access-rngh8\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.448893 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.448851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46da0e33-7e1d-45f2-ba29-4f9e225abccb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.549782 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.549746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngh8\" (UniqueName: \"kubernetes.io/projected/46da0e33-7e1d-45f2-ba29-4f9e225abccb-kube-api-access-rngh8\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.549972 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.549786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46da0e33-7e1d-45f2-ba29-4f9e225abccb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.549972 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.549863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.549972 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:41.549966    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-serving-cert: secret "maas-api-e2e-shared-a-fbed9b-serving-cert" not found
Jul 16 22:41:41.550117 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:41.550022    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls podName:46da0e33-7e1d-45f2-ba29-4f9e225abccb nodeName:}" failed. No retries permitted until 2026-07-16 22:41:42.050006345 +0000 UTC m=+3284.831904171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls") pod "maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" (UID: "46da0e33-7e1d-45f2-ba29-4f9e225abccb") : secret "maas-api-e2e-shared-a-fbed9b-serving-cert" not found
Jul 16 22:41:41.550702 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.550673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46da0e33-7e1d-45f2-ba29-4f9e225abccb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.590498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.590463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngh8\" (UniqueName: \"kubernetes.io/projected/46da0e33-7e1d-45f2-ba29-4f9e225abccb-kube-api-access-rngh8\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:41.655514 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.655484    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:41.848713 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:41.848679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:42.054433 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.054332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:42.057012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.056980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls\") pod \"maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:42.102799 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.102762    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerID="5ea1dde8cef30612a8002165fbe4e637173919244298e1013ab9104e088e0da4" exitCode=6
Jul 16 22:41:42.102961 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.102836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" event={"ID":"d1d79126-d679-4616-968b-a2bed6ddba7d","Type":"ContainerDied","Data":"5ea1dde8cef30612a8002165fbe4e637173919244298e1013ab9104e088e0da4"}
Jul 16 22:41:42.102961 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.102897    2562 scope.go:117] "RemoveContainer" containerID="4433a04b93b87bf346cbc06daa76232f4d10d07638d76abe7c0221cc46b16193"
Jul 16 22:41:42.103225 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.103201    2562 scope.go:117] "RemoveContainer" containerID="5ea1dde8cef30612a8002165fbe4e637173919244298e1013ab9104e088e0da4"
Jul 16 22:41:42.103478 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:42.103452    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-hh8r9_odh-ai-gateway-infra(d1d79126-d679-4616-968b-a2bed6ddba7d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d"
Jul 16 22:41:42.124510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.124481    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"]
Jul 16 22:41:42.316807 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.316710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:42.361827 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.361786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:42.466290 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.466249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"]
Jul 16 22:41:42.466485 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:41:42.466463    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46da0e33_7e1d_45f2_ba29_4f9e225abccb.slice/crio-bcec604834db0f610800e0c66cc07dfb12c922bd606ab7b34618597e0b61a2b9 WatchSource:0}: Error finding container bcec604834db0f610800e0c66cc07dfb12c922bd606ab7b34618597e0b61a2b9: Status 404 returned error can't find the container with id bcec604834db0f610800e0c66cc07dfb12c922bd606ab7b34618597e0b61a2b9
Jul 16 22:41:42.473525 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:42.473500    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:43.110971 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.110944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" event={"ID":"46da0e33-7e1d-45f2-ba29-4f9e225abccb","Type":"ContainerStarted","Data":"42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e"}
Jul 16 22:41:43.111270 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.110991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" event={"ID":"46da0e33-7e1d-45f2-ba29-4f9e225abccb","Type":"ContainerStarted","Data":"bcec604834db0f610800e0c66cc07dfb12c922bd606ab7b34618597e0b61a2b9"}
Jul 16 22:41:43.111270 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.111015    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:43.149380 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.149089    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" podStartSLOduration=1.8052305990000002 podStartE2EDuration="2.149069051s" totalImagesPullingTime="343.838452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:43.148908831 +0000 UTC m=+3285.930806679" watchObservedRunningTime="2026-07-16 22:41:43.149069051 +0000 UTC m=+3285.930966899"
Jul 16 22:41:43.266632 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.266606    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:43.366451 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.366383    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d79126-d679-4616-968b-a2bed6ddba7d-maas-api-service-ca\") pod \"d1d79126-d679-4616-968b-a2bed6ddba7d\" (UID: \"d1d79126-d679-4616-968b-a2bed6ddba7d\") "
Jul 16 22:41:43.366762 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.366739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d79126-d679-4616-968b-a2bed6ddba7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1d79126-d679-4616-968b-a2bed6ddba7d" (UID: "d1d79126-d679-4616-968b-a2bed6ddba7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:43.467513 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:43.467482    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d79126-d679-4616-968b-a2bed6ddba7d-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:41:44.117235 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:44.117134    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9" event={"ID":"d1d79126-d679-4616-968b-a2bed6ddba7d","Type":"ContainerDied","Data":"6a917b8fbf244e21262dc12931f1a8d38450e24dff1c75b98e1fe4ff59a1c477"}
Jul 16 22:41:44.117235 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:44.117146    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"
Jul 16 22:41:44.117785 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:44.117216    2562 scope.go:117] "RemoveContainer" containerID="5ea1dde8cef30612a8002165fbe4e637173919244298e1013ab9104e088e0da4"
Jul 16 22:41:44.145822 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:44.145794    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"]
Jul 16 22:41:44.148994 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:44.148969    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-hh8r9"]
Jul 16 22:41:45.646292 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:45.646251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:45.750596 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:45.750563    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" path="/var/lib/kubelet/pods/d1d79126-d679-4616-968b-a2bed6ddba7d/volumes"
Jul 16 22:41:45.991755 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:45.991725    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:46.325590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:46.325489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:46.415663 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:46.415616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:49.123191 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.123157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:41:49.209000 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.208966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:49.315041 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.315009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:49.315999 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.315976    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"]
Jul 16 22:41:49.316333 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316321    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316335    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316344    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316349    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316371    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316492 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316378    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316492 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316427    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316492 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316434    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:41:49.316492 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.316442    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28257386-a939-4549-9e4c-42c9bfaf9848" containerName="revoke-keys"
Jul 16 22:41:49.320561 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.320529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.323268 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.323248    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-106c85-serving-cert\""
Jul 16 22:41:49.331242 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.331217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"]
Jul 16 22:41:49.422840 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.422808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.423060 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.422863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-openshift-service-ca\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.423060 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.422963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdlv\" (UniqueName: \"kubernetes.io/projected/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-kube-api-access-qrdlv\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.433058 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.433019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:49.523704 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.523670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-openshift-service-ca\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.523867 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.523717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdlv\" (UniqueName: \"kubernetes.io/projected/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-kube-api-access-qrdlv\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.523867 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.523789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.523938 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:49.523901    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-serving-cert: secret "maas-api-e2e-shared-b-106c85-serving-cert" not found
Jul 16 22:41:49.524001 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:41:49.523989    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls podName:aaeb5cbd-d791-4f13-8e5c-48c77dc4d077 nodeName:}" failed. No retries permitted until 2026-07-16 22:41:50.02396608 +0000 UTC m=+3292.805863907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls") pod "maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" (UID: "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077") : secret "maas-api-e2e-shared-b-106c85-serving-cert" not found
Jul 16 22:41:49.524448 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.524429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-openshift-service-ca\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.550357 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.550326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdlv\" (UniqueName: \"kubernetes.io/projected/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-kube-api-access-qrdlv\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:49.855042 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:49.854941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:50.026774 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:50.026742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:50.029535 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:50.029507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls\") pod \"maas-api-e2e-shared-b-106c85-855ddbff65-b7bps\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:50.231681 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:50.231648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:50.322454 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:50.322422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:50.393885 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:50.393857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"]
Jul 16 22:41:50.397376 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:41:50.397349    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaeb5cbd_d791_4f13_8e5c_48c77dc4d077.slice/crio-e2dffc141ac59070ab20e6f9854a83d489efdaf3816cf7d9891f1856c5bf584d WatchSource:0}: Error finding container e2dffc141ac59070ab20e6f9854a83d489efdaf3816cf7d9891f1856c5bf584d: Status 404 returned error can't find the container with id e2dffc141ac59070ab20e6f9854a83d489efdaf3816cf7d9891f1856c5bf584d
Jul 16 22:41:50.459770 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:50.459739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:51.153105 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:51.153066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" event={"ID":"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077","Type":"ContainerStarted","Data":"f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25"}
Jul 16 22:41:51.153280 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:51.153120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" event={"ID":"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077","Type":"ContainerStarted","Data":"e2dffc141ac59070ab20e6f9854a83d489efdaf3816cf7d9891f1856c5bf584d"}
Jul 16 22:41:51.153280 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:51.153191    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:41:51.172390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:51.172337    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" podStartSLOduration=1.835250803 podStartE2EDuration="2.172321429s" totalImagesPullingTime="337.070626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:51.170268754 +0000 UTC m=+3293.952166614" watchObservedRunningTime="2026-07-16 22:41:51.172321429 +0000 UTC m=+3293.954219334"
Jul 16 22:41:53.711199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:53.711162    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:57.163234 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:41:57.163207    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:42:03.233189 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:03.233153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:03.329390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:03.329354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:07.562988 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.562954    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"]
Jul 16 22:42:07.563390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.563378    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d79126-d679-4616-968b-a2bed6ddba7d" containerName="revoke-keys"
Jul 16 22:42:07.567838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.567823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:07.574616 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.574285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:42:07.576805 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.576778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"]
Jul 16 22:42:07.688544 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.688510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f12b7-ed5a-446c-b570-318d0feb9d13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675\" (UID: \"2d1f12b7-ed5a-446c-b570-318d0feb9d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:07.789651 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.789616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f12b7-ed5a-446c-b570-318d0feb9d13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675\" (UID: \"2d1f12b7-ed5a-446c-b570-318d0feb9d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:07.790263 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.790239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f12b7-ed5a-446c-b570-318d0feb9d13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675\" (UID: \"2d1f12b7-ed5a-446c-b570-318d0feb9d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:07.879297 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:07.879211    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:08.016338 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:08.016303    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"]
Jul 16 22:42:08.022176 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:42:08.022134    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1f12b7_ed5a_446c_b570_318d0feb9d13.slice/crio-d49c1dc971bf1df4dbe12b64fb5dbee4b4fcae054cc8c0bf27b77a93ec3430b5 WatchSource:0}: Error finding container d49c1dc971bf1df4dbe12b64fb5dbee4b4fcae054cc8c0bf27b77a93ec3430b5: Status 404 returned error can't find the container with id d49c1dc971bf1df4dbe12b64fb5dbee4b4fcae054cc8c0bf27b77a93ec3430b5
Jul 16 22:42:08.235781 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:08.235745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" event={"ID":"2d1f12b7-ed5a-446c-b570-318d0feb9d13","Type":"ContainerStarted","Data":"d49c1dc971bf1df4dbe12b64fb5dbee4b4fcae054cc8c0bf27b77a93ec3430b5"}
Jul 16 22:42:09.241800 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:09.241760    2562 generic.go:356] "Generic (PLEG): container finished" podID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerID="1fbcce8714da9b468f4717fe5d705c1bce3d65c4250efdc8b3c6935e1d2d50a5" exitCode=6
Jul 16 22:42:09.242238 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:09.241845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" event={"ID":"2d1f12b7-ed5a-446c-b570-318d0feb9d13","Type":"ContainerDied","Data":"1fbcce8714da9b468f4717fe5d705c1bce3d65c4250efdc8b3c6935e1d2d50a5"}
Jul 16 22:42:09.242238 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:09.242051    2562 scope.go:117] "RemoveContainer" containerID="1fbcce8714da9b468f4717fe5d705c1bce3d65c4250efdc8b3c6935e1d2d50a5"
Jul 16 22:42:09.350829 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:09.350795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:10.248328 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:10.248287    2562 generic.go:356] "Generic (PLEG): container finished" podID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerID="a393f30685b3374298a4c8c3f19711f9492ef9d34bdeef767b44c1fdf40e4ef3" exitCode=6
Jul 16 22:42:10.248826 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:10.248379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" event={"ID":"2d1f12b7-ed5a-446c-b570-318d0feb9d13","Type":"ContainerDied","Data":"a393f30685b3374298a4c8c3f19711f9492ef9d34bdeef767b44c1fdf40e4ef3"}
Jul 16 22:42:10.248826 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:10.248461    2562 scope.go:117] "RemoveContainer" containerID="1fbcce8714da9b468f4717fe5d705c1bce3d65c4250efdc8b3c6935e1d2d50a5"
Jul 16 22:42:10.248826 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:10.248664    2562 scope.go:117] "RemoveContainer" containerID="a393f30685b3374298a4c8c3f19711f9492ef9d34bdeef767b44c1fdf40e4ef3"
Jul 16 22:42:10.248959 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:10.248846    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675_odh-ai-gateway-infra(2d1f12b7-ed5a-446c-b570-318d0feb9d13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13"
Jul 16 22:42:11.255482 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:11.255455    2562 scope.go:117] "RemoveContainer" containerID="a393f30685b3374298a4c8c3f19711f9492ef9d34bdeef767b44c1fdf40e4ef3"
Jul 16 22:42:11.255920 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:11.255683    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675_odh-ai-gateway-infra(2d1f12b7-ed5a-446c-b570-318d0feb9d13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13"
Jul 16 22:42:11.324891 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:11.324859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:11.426743 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:11.426712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:21.745664 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:21.745586    2562 scope.go:117] "RemoveContainer" containerID="a393f30685b3374298a4c8c3f19711f9492ef9d34bdeef767b44c1fdf40e4ef3"
Jul 16 22:42:22.308315 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:22.308225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" event={"ID":"2d1f12b7-ed5a-446c-b570-318d0feb9d13","Type":"ContainerStarted","Data":"fa34de4e444253c871c6b761c9921de1e37955571c782fa2f7551e1a4ef6da78"}
Jul 16 22:42:22.328904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:22.328799    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" podStartSLOduration=15.328779047 podStartE2EDuration="15.328779047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:22.324509414 +0000 UTC m=+3325.106407265" watchObservedRunningTime="2026-07-16 22:42:22.328779047 +0000 UTC m=+3325.110676899"
Jul 16 22:42:23.314094 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.314058    2562 generic.go:356] "Generic (PLEG): container finished" podID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerID="fa34de4e444253c871c6b761c9921de1e37955571c782fa2f7551e1a4ef6da78" exitCode=6
Jul 16 22:42:23.314597 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.314115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" event={"ID":"2d1f12b7-ed5a-446c-b570-318d0feb9d13","Type":"ContainerDied","Data":"fa34de4e444253c871c6b761c9921de1e37955571c782fa2f7551e1a4ef6da78"}
Jul 16 22:42:23.314597 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.314175    2562 scope.go:117] "RemoveContainer" containerID="a393f30685b3374298a4c8c3f19711f9492ef9d34bdeef767b44c1fdf40e4ef3"
Jul 16 22:42:23.314597 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.314509    2562 scope.go:117] "RemoveContainer" containerID="fa34de4e444253c871c6b761c9921de1e37955571c782fa2f7551e1a4ef6da78"
Jul 16 22:42:23.314806 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:23.314786    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675_odh-ai-gateway-infra(2d1f12b7-ed5a-446c-b570-318d0feb9d13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13"
Jul 16 22:42:23.343921 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.343894    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"]
Jul 16 22:42:23.630584 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.630489    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"]
Jul 16 22:42:23.635007 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.634990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:23.644123 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.644099    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"]
Jul 16 22:42:23.723772 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.723728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9527548-47a0-4aa2-827b-04d7b5f5ae1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-wc6rq\" (UID: \"d9527548-47a0-4aa2-827b-04d7b5f5ae1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:23.824341 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.824311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9527548-47a0-4aa2-827b-04d7b5f5ae1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-wc6rq\" (UID: \"d9527548-47a0-4aa2-827b-04d7b5f5ae1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:23.824986 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.824967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9527548-47a0-4aa2-827b-04d7b5f5ae1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-wc6rq\" (UID: \"d9527548-47a0-4aa2-827b-04d7b5f5ae1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:23.946603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:23.946575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:24.084327 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.084290    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"]
Jul 16 22:42:24.087375 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:42:24.087347    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9527548_47a0_4aa2_827b_04d7b5f5ae1e.slice/crio-e9819c99d55fa731922d0dd1b93fce857fd942538ce521c6a5fc46c033d42e0a WatchSource:0}: Error finding container e9819c99d55fa731922d0dd1b93fce857fd942538ce521c6a5fc46c033d42e0a: Status 404 returned error can't find the container with id e9819c99d55fa731922d0dd1b93fce857fd942538ce521c6a5fc46c033d42e0a
Jul 16 22:42:24.322029 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.321991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerStarted","Data":"aec87e00a5d03e23fe46b874be761251dc8b458d1e29faf1ca6f3fabda8ba629"}
Jul 16 22:42:24.322413 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.322061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerStarted","Data":"e9819c99d55fa731922d0dd1b93fce857fd942538ce521c6a5fc46c033d42e0a"}
Jul 16 22:42:24.441613 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.441589    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:24.462986 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.462936    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" podStartSLOduration=1.462921863 podStartE2EDuration="1.462921863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:24.372250974 +0000 UTC m=+3327.154148825" watchObservedRunningTime="2026-07-16 22:42:24.462921863 +0000 UTC m=+3327.244819728"
Jul 16 22:42:24.531785 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.531698    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f12b7-ed5a-446c-b570-318d0feb9d13-maas-api-service-ca\") pod \"2d1f12b7-ed5a-446c-b570-318d0feb9d13\" (UID: \"2d1f12b7-ed5a-446c-b570-318d0feb9d13\") "
Jul 16 22:42:24.532064 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.532040    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1f12b7-ed5a-446c-b570-318d0feb9d13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d1f12b7-ed5a-446c-b570-318d0feb9d13" (UID: "2d1f12b7-ed5a-446c-b570-318d0feb9d13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:24.632417 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:24.632379    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1f12b7-ed5a-446c-b570-318d0feb9d13-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:42:25.334247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.334159    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"
Jul 16 22:42:25.334247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.334151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675" event={"ID":"2d1f12b7-ed5a-446c-b570-318d0feb9d13","Type":"ContainerDied","Data":"d49c1dc971bf1df4dbe12b64fb5dbee4b4fcae054cc8c0bf27b77a93ec3430b5"}
Jul 16 22:42:25.334789 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.334316    2562 scope.go:117] "RemoveContainer" containerID="fa34de4e444253c871c6b761c9921de1e37955571c782fa2f7551e1a4ef6da78"
Jul 16 22:42:25.335983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.335923    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerID="aec87e00a5d03e23fe46b874be761251dc8b458d1e29faf1ca6f3fabda8ba629" exitCode=6
Jul 16 22:42:25.335983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.336005    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerDied","Data":"aec87e00a5d03e23fe46b874be761251dc8b458d1e29faf1ca6f3fabda8ba629"}
Jul 16 22:42:25.336189 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.336173    2562 scope.go:117] "RemoveContainer" containerID="aec87e00a5d03e23fe46b874be761251dc8b458d1e29faf1ca6f3fabda8ba629"
Jul 16 22:42:25.370208 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.370180    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"]
Jul 16 22:42:25.375346 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.375325    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-vr675"]
Jul 16 22:42:25.751823 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:25.751787    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" path="/var/lib/kubelet/pods/2d1f12b7-ed5a-446c-b570-318d0feb9d13/volumes"
Jul 16 22:42:26.343746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:26.343712    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerID="f04e4ecfdd6c6e20d36c7091185ab1bd0cd99bd2c9c790066eda062186cd7e36" exitCode=6
Jul 16 22:42:26.344205 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:26.343796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerDied","Data":"f04e4ecfdd6c6e20d36c7091185ab1bd0cd99bd2c9c790066eda062186cd7e36"}
Jul 16 22:42:26.344205 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:26.343859    2562 scope.go:117] "RemoveContainer" containerID="aec87e00a5d03e23fe46b874be761251dc8b458d1e29faf1ca6f3fabda8ba629"
Jul 16 22:42:26.344205 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:26.344127    2562 scope.go:117] "RemoveContainer" containerID="f04e4ecfdd6c6e20d36c7091185ab1bd0cd99bd2c9c790066eda062186cd7e36"
Jul 16 22:42:26.344380 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:26.344354    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-wc6rq_odh-ai-gateway-infra(d9527548-47a0-4aa2-827b-04d7b5f5ae1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e"
Jul 16 22:42:27.351027 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:27.350998    2562 scope.go:117] "RemoveContainer" containerID="f04e4ecfdd6c6e20d36c7091185ab1bd0cd99bd2c9c790066eda062186cd7e36"
Jul 16 22:42:27.351472 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:27.351255    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-wc6rq_odh-ai-gateway-infra(d9527548-47a0-4aa2-827b-04d7b5f5ae1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e"
Jul 16 22:42:30.615312 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615281    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"]
Jul 16 22:42:30.615705 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615689    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615709    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615733    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615738    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615748    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615753    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615812    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615819    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.615952 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.615826    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1f12b7-ed5a-446c-b570-318d0feb9d13" containerName="revoke-keys"
Jul 16 22:42:30.620074 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.620054    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.622793 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.622770    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"kube-root-ca.crt\""
Jul 16 22:42:30.623766 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.623747    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"default-dockercfg-25fwx\""
Jul 16 22:42:30.623877 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.623794    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"openshift-service-ca.crt\""
Jul 16 22:42:30.623945 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.623907    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"auth-test-model-106c85-kserve-self-signed-certs\""
Jul 16 22:42:30.632779 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.632756    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"]
Jul 16 22:42:30.655788 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.655761    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:30.685935 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.685901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-model-cache\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.686097 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.685954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-home\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.686097 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.685978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-dshm\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.686097 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.686038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwvs\" (UniqueName: \"kubernetes.io/projected/776d6934-2835-41c5-8388-11b3757217ef-kube-api-access-xqwvs\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.686097 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.686075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-tmp-dir\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.686272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.686185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776d6934-2835-41c5-8388-11b3757217ef-tls-certs\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.786640 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.786606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-tmp-dir\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.786836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.786683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776d6934-2835-41c5-8388-11b3757217ef-tls-certs\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.786836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.786725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-model-cache\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.786836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.786762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-home\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.786836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.786785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-dshm\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.787049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.786879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwvs\" (UniqueName: \"kubernetes.io/projected/776d6934-2835-41c5-8388-11b3757217ef-kube-api-access-xqwvs\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.787204 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.787172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-model-cache\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.787339 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.787221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-home\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.787339 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.787299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-tmp-dir\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.789249 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.789225    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-dshm\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.789420 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.789403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776d6934-2835-41c5-8388-11b3757217ef-tls-certs\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.796492 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.796467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwvs\" (UniqueName: \"kubernetes.io/projected/776d6934-2835-41c5-8388-11b3757217ef-kube-api-access-xqwvs\") pod \"auth-test-model-106c85-kserve-676b84c6c5-g7vth\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") " pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:30.929199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:30.929169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:31.065429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:31.065406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"]
Jul 16 22:42:31.067936 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:42:31.067898    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776d6934_2835_41c5_8388_11b3757217ef.slice/crio-b937f492d2e8c38915f31b6ac95f4e97f4d79d7cf88d3357d031cb1af65ccc91 WatchSource:0}: Error finding container b937f492d2e8c38915f31b6ac95f4e97f4d79d7cf88d3357d031cb1af65ccc91: Status 404 returned error can't find the container with id b937f492d2e8c38915f31b6ac95f4e97f4d79d7cf88d3357d031cb1af65ccc91
Jul 16 22:42:31.370760 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:31.370664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth" event={"ID":"776d6934-2835-41c5-8388-11b3757217ef","Type":"ContainerStarted","Data":"7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437"}
Jul 16 22:42:31.370760 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:31.370717    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth" event={"ID":"776d6934-2835-41c5-8388-11b3757217ef","Type":"ContainerStarted","Data":"b937f492d2e8c38915f31b6ac95f4e97f4d79d7cf88d3357d031cb1af65ccc91"}
Jul 16 22:42:31.389937 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:31.389883    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth" podStartSLOduration=1.389864222 podStartE2EDuration="1.389864222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:31.389150685 +0000 UTC m=+3334.171048536" watchObservedRunningTime="2026-07-16 22:42:31.389864222 +0000 UTC m=+3334.171762071"
Jul 16 22:42:40.745945 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:40.745908    2562 scope.go:117] "RemoveContainer" containerID="f04e4ecfdd6c6e20d36c7091185ab1bd0cd99bd2c9c790066eda062186cd7e36"
Jul 16 22:42:40.930133 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:40.930097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:40.930322 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:40.930150    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:40.942651 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:40.942625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:41.418220 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:41.418185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerStarted","Data":"5cbbae91197e1b3f45d812e41e8fc7b0c1f05c6294c38bcae8d6288bd9bd6a26"}
Jul 16 22:42:41.430014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:41.429984    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:42:42.424910 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:42.424881    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerID="5cbbae91197e1b3f45d812e41e8fc7b0c1f05c6294c38bcae8d6288bd9bd6a26" exitCode=6
Jul 16 22:42:42.425343 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:42.424959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerDied","Data":"5cbbae91197e1b3f45d812e41e8fc7b0c1f05c6294c38bcae8d6288bd9bd6a26"}
Jul 16 22:42:42.425343 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:42.425030    2562 scope.go:117] "RemoveContainer" containerID="f04e4ecfdd6c6e20d36c7091185ab1bd0cd99bd2c9c790066eda062186cd7e36"
Jul 16 22:42:42.425477 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:42.425462    2562 scope.go:117] "RemoveContainer" containerID="5cbbae91197e1b3f45d812e41e8fc7b0c1f05c6294c38bcae8d6288bd9bd6a26"
Jul 16 22:42:42.425717 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:42.425697    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-wc6rq_odh-ai-gateway-infra(d9527548-47a0-4aa2-827b-04d7b5f5ae1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e"
Jul 16 22:42:42.446203 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:42.446166    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"]
Jul 16 22:42:43.566375 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:43.566345    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:43.709002 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:43.708899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9527548-47a0-4aa2-827b-04d7b5f5ae1e-maas-api-service-ca\") pod \"d9527548-47a0-4aa2-827b-04d7b5f5ae1e\" (UID: \"d9527548-47a0-4aa2-827b-04d7b5f5ae1e\") "
Jul 16 22:42:43.709275 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:43.709253    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9527548-47a0-4aa2-827b-04d7b5f5ae1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9527548-47a0-4aa2-827b-04d7b5f5ae1e" (UID: "d9527548-47a0-4aa2-827b-04d7b5f5ae1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:43.809820 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:43.809783    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9527548-47a0-4aa2-827b-04d7b5f5ae1e-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:42:44.437182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:44.437150    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"
Jul 16 22:42:44.437373 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:44.437199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq" event={"ID":"d9527548-47a0-4aa2-827b-04d7b5f5ae1e","Type":"ContainerDied","Data":"e9819c99d55fa731922d0dd1b93fce857fd942538ce521c6a5fc46c033d42e0a"}
Jul 16 22:42:44.437373 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:44.437259    2562 scope.go:117] "RemoveContainer" containerID="5cbbae91197e1b3f45d812e41e8fc7b0c1f05c6294c38bcae8d6288bd9bd6a26"
Jul 16 22:42:44.464180 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:44.464152    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"]
Jul 16 22:42:44.472536 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:44.472515    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-wc6rq"]
Jul 16 22:42:45.751714 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:45.751679    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" path="/var/lib/kubelet/pods/d9527548-47a0-4aa2-827b-04d7b5f5ae1e/volumes"
Jul 16 22:42:46.046520 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"]
Jul 16 22:42:46.046808 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046794    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.046856 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046809    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.046856 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046829    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.046856 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046834    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.046951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046891    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.046951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046898    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.046951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.046905    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:46.051206 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.051189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:42:46.055436 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.055415    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:42:46.073333 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.073302    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"]
Jul 16 22:42:46.130169 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.130139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb4a619-f791-40bf-bf73-61823f55cbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs\" (UID: \"7bb4a619-f791-40bf-bf73-61823f55cbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:42:46.231246 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.231203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb4a619-f791-40bf-bf73-61823f55cbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs\" (UID: \"7bb4a619-f791-40bf-bf73-61823f55cbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:42:46.231872 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.231853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb4a619-f791-40bf-bf73-61823f55cbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs\" (UID: \"7bb4a619-f791-40bf-bf73-61823f55cbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:42:46.361380 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.361298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:42:46.505239 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:46.505218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"]
Jul 16 22:42:46.507918 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:42:46.507891    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb4a619_f791_40bf_bf73_61823f55cbe7.slice/crio-82fb39f4b7e5edee53cbf74055a4730e3bbce187df2f3b26d59e705ab279c846 WatchSource:0}: Error finding container 82fb39f4b7e5edee53cbf74055a4730e3bbce187df2f3b26d59e705ab279c846: Status 404 returned error can't find the container with id 82fb39f4b7e5edee53cbf74055a4730e3bbce187df2f3b26d59e705ab279c846
Jul 16 22:42:47.456531 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:47.456428    2562 generic.go:356] "Generic (PLEG): container finished" podID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerID="456b6c47f0684289f6db9a4ac9817d4baf21af38568e24e25bf1f6cb05489336" exitCode=6
Jul 16 22:42:47.456949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:47.456516    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" event={"ID":"7bb4a619-f791-40bf-bf73-61823f55cbe7","Type":"ContainerDied","Data":"456b6c47f0684289f6db9a4ac9817d4baf21af38568e24e25bf1f6cb05489336"}
Jul 16 22:42:47.456949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:47.456642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" event={"ID":"7bb4a619-f791-40bf-bf73-61823f55cbe7","Type":"ContainerStarted","Data":"82fb39f4b7e5edee53cbf74055a4730e3bbce187df2f3b26d59e705ab279c846"}
Jul 16 22:42:47.456949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:47.456734    2562 scope.go:117] "RemoveContainer" containerID="456b6c47f0684289f6db9a4ac9817d4baf21af38568e24e25bf1f6cb05489336"
Jul 16 22:42:48.463034 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:48.463003    2562 generic.go:356] "Generic (PLEG): container finished" podID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerID="407132d0f012e5960ae64ab4868d78cdb9a4c8ea4f9120adeee4dfc10af70403" exitCode=6
Jul 16 22:42:48.463445 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:48.463042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" event={"ID":"7bb4a619-f791-40bf-bf73-61823f55cbe7","Type":"ContainerDied","Data":"407132d0f012e5960ae64ab4868d78cdb9a4c8ea4f9120adeee4dfc10af70403"}
Jul 16 22:42:48.463445 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:48.463130    2562 scope.go:117] "RemoveContainer" containerID="456b6c47f0684289f6db9a4ac9817d4baf21af38568e24e25bf1f6cb05489336"
Jul 16 22:42:48.463445 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:48.463326    2562 scope.go:117] "RemoveContainer" containerID="407132d0f012e5960ae64ab4868d78cdb9a4c8ea4f9120adeee4dfc10af70403"
Jul 16 22:42:48.463644 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:48.463584    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs_odh-ai-gateway-infra(7bb4a619-f791-40bf-bf73-61823f55cbe7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7"
Jul 16 22:42:49.469628 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:49.469599    2562 scope.go:117] "RemoveContainer" containerID="407132d0f012e5960ae64ab4868d78cdb9a4c8ea4f9120adeee4dfc10af70403"
Jul 16 22:42:49.470006 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:42:49.469814    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs_odh-ai-gateway-infra(7bb4a619-f791-40bf-bf73-61823f55cbe7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7"
Jul 16 22:42:57.604529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.604502    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"]
Jul 16 22:42:57.604908 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.604876    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:57.604908 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.604887    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9527548-47a0-4aa2-827b-04d7b5f5ae1e" containerName="revoke-keys"
Jul 16 22:42:57.609274 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.609256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:42:57.616454 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.616428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"]
Jul 16 22:42:57.736155 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.736116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3be5072-ee7d-4cd1-80e6-79eea0ae843b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz\" (UID: \"c3be5072-ee7d-4cd1-80e6-79eea0ae843b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:42:57.837317 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.837285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3be5072-ee7d-4cd1-80e6-79eea0ae843b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz\" (UID: \"c3be5072-ee7d-4cd1-80e6-79eea0ae843b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:42:57.837961 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.837940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3be5072-ee7d-4cd1-80e6-79eea0ae843b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz\" (UID: \"c3be5072-ee7d-4cd1-80e6-79eea0ae843b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:42:57.921256 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:57.921210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:42:58.062247 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:58.062211    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"]
Jul 16 22:42:58.068903 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:42:58.068874    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3be5072_ee7d_4cd1_80e6_79eea0ae843b.slice/crio-2db2566846e27a74e1675ff863ab1b1f5daf9847e1290dcb4eef021cbf159e58 WatchSource:0}: Error finding container 2db2566846e27a74e1675ff863ab1b1f5daf9847e1290dcb4eef021cbf159e58: Status 404 returned error can't find the container with id 2db2566846e27a74e1675ff863ab1b1f5daf9847e1290dcb4eef021cbf159e58
Jul 16 22:42:58.513006 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:58.512900    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" event={"ID":"c3be5072-ee7d-4cd1-80e6-79eea0ae843b","Type":"ContainerStarted","Data":"43587449d789736190b1abbc2778a281eb8385ef48ad35608e8e9918ec9407b1"}
Jul 16 22:42:58.513006 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:58.512990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" event={"ID":"c3be5072-ee7d-4cd1-80e6-79eea0ae843b","Type":"ContainerStarted","Data":"2db2566846e27a74e1675ff863ab1b1f5daf9847e1290dcb4eef021cbf159e58"}
Jul 16 22:42:59.519458 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:59.519422    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerID="43587449d789736190b1abbc2778a281eb8385ef48ad35608e8e9918ec9407b1" exitCode=6
Jul 16 22:42:59.519951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:59.519505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" event={"ID":"c3be5072-ee7d-4cd1-80e6-79eea0ae843b","Type":"ContainerDied","Data":"43587449d789736190b1abbc2778a281eb8385ef48ad35608e8e9918ec9407b1"}
Jul 16 22:42:59.519951 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:42:59.519782    2562 scope.go:117] "RemoveContainer" containerID="43587449d789736190b1abbc2778a281eb8385ef48ad35608e8e9918ec9407b1"
Jul 16 22:43:00.526086 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:00.526056    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerID="baad93c0a6eb3a46ad857a049da6cc9a5f1c672f6aeecc136a0b1b9f4cb47317" exitCode=6
Jul 16 22:43:00.526479 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:00.526142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" event={"ID":"c3be5072-ee7d-4cd1-80e6-79eea0ae843b","Type":"ContainerDied","Data":"baad93c0a6eb3a46ad857a049da6cc9a5f1c672f6aeecc136a0b1b9f4cb47317"}
Jul 16 22:43:00.526479 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:00.526205    2562 scope.go:117] "RemoveContainer" containerID="43587449d789736190b1abbc2778a281eb8385ef48ad35608e8e9918ec9407b1"
Jul 16 22:43:00.526479 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:00.526404    2562 scope.go:117] "RemoveContainer" containerID="baad93c0a6eb3a46ad857a049da6cc9a5f1c672f6aeecc136a0b1b9f4cb47317"
Jul 16 22:43:00.526681 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:00.526663    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz_odh-ai-gateway-infra(c3be5072-ee7d-4cd1-80e6-79eea0ae843b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b"
Jul 16 22:43:01.532800 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:01.532774    2562 scope.go:117] "RemoveContainer" containerID="baad93c0a6eb3a46ad857a049da6cc9a5f1c672f6aeecc136a0b1b9f4cb47317"
Jul 16 22:43:01.533180 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:01.532966    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz_odh-ai-gateway-infra(c3be5072-ee7d-4cd1-80e6-79eea0ae843b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b"
Jul 16 22:43:04.745357 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:04.745326    2562 scope.go:117] "RemoveContainer" containerID="407132d0f012e5960ae64ab4868d78cdb9a4c8ea4f9120adeee4dfc10af70403"
Jul 16 22:43:05.554155 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:05.554120    2562 generic.go:356] "Generic (PLEG): container finished" podID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerID="a1709d534c300a13c16bf8ccced799e7f9b4c2054a0a8b6a5521e538cc5052ce" exitCode=6
Jul 16 22:43:05.554415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:05.554202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" event={"ID":"7bb4a619-f791-40bf-bf73-61823f55cbe7","Type":"ContainerDied","Data":"a1709d534c300a13c16bf8ccced799e7f9b4c2054a0a8b6a5521e538cc5052ce"}
Jul 16 22:43:05.554415 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:05.554311    2562 scope.go:117] "RemoveContainer" containerID="407132d0f012e5960ae64ab4868d78cdb9a4c8ea4f9120adeee4dfc10af70403"
Jul 16 22:43:05.554664 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:05.554648    2562 scope.go:117] "RemoveContainer" containerID="a1709d534c300a13c16bf8ccced799e7f9b4c2054a0a8b6a5521e538cc5052ce"
Jul 16 22:43:05.554922 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:05.554899    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs_odh-ai-gateway-infra(7bb4a619-f791-40bf-bf73-61823f55cbe7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7"
Jul 16 22:43:06.585986 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:06.585949    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"]
Jul 16 22:43:06.711374 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:06.711349    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:43:06.814123 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:06.814093    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb4a619-f791-40bf-bf73-61823f55cbe7-maas-api-service-ca\") pod \"7bb4a619-f791-40bf-bf73-61823f55cbe7\" (UID: \"7bb4a619-f791-40bf-bf73-61823f55cbe7\") "
Jul 16 22:43:06.814483 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:06.814456    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb4a619-f791-40bf-bf73-61823f55cbe7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bb4a619-f791-40bf-bf73-61823f55cbe7" (UID: "7bb4a619-f791-40bf-bf73-61823f55cbe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:06.915518 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:06.915487    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb4a619-f791-40bf-bf73-61823f55cbe7-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:43:07.566450 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:07.566419    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"
Jul 16 22:43:07.566630 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:07.566451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs" event={"ID":"7bb4a619-f791-40bf-bf73-61823f55cbe7","Type":"ContainerDied","Data":"82fb39f4b7e5edee53cbf74055a4730e3bbce187df2f3b26d59e705ab279c846"}
Jul 16 22:43:07.566630 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:07.566512    2562 scope.go:117] "RemoveContainer" containerID="a1709d534c300a13c16bf8ccced799e7f9b4c2054a0a8b6a5521e538cc5052ce"
Jul 16 22:43:07.588207 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:07.588180    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"]
Jul 16 22:43:07.593043 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:07.593022    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-ktzzs"]
Jul 16 22:43:07.750733 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:07.750697    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" path="/var/lib/kubelet/pods/7bb4a619-f791-40bf-bf73-61823f55cbe7/volumes"
Jul 16 22:43:12.483704 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:12.483662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:15.745543 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:15.745508    2562 scope.go:117] "RemoveContainer" containerID="baad93c0a6eb3a46ad857a049da6cc9a5f1c672f6aeecc136a0b1b9f4cb47317"
Jul 16 22:43:16.610177 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:16.610147    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerID="66f3acce2426af4a83d6924dcb78cecc0e9635d95fac05e72518039cc597a511" exitCode=6
Jul 16 22:43:16.610352 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:16.610188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" event={"ID":"c3be5072-ee7d-4cd1-80e6-79eea0ae843b","Type":"ContainerDied","Data":"66f3acce2426af4a83d6924dcb78cecc0e9635d95fac05e72518039cc597a511"}
Jul 16 22:43:16.610352 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:16.610243    2562 scope.go:117] "RemoveContainer" containerID="baad93c0a6eb3a46ad857a049da6cc9a5f1c672f6aeecc136a0b1b9f4cb47317"
Jul 16 22:43:16.610636 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:16.610616    2562 scope.go:117] "RemoveContainer" containerID="66f3acce2426af4a83d6924dcb78cecc0e9635d95fac05e72518039cc597a511"
Jul 16 22:43:16.610881 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:16.610854    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz_odh-ai-gateway-infra(c3be5072-ee7d-4cd1-80e6-79eea0ae843b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b"
Jul 16 22:43:17.648008 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:17.647968    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"]
Jul 16 22:43:17.790109 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:17.790087    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:43:17.910493 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:17.910467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3be5072-ee7d-4cd1-80e6-79eea0ae843b-maas-api-service-ca\") pod \"c3be5072-ee7d-4cd1-80e6-79eea0ae843b\" (UID: \"c3be5072-ee7d-4cd1-80e6-79eea0ae843b\") "
Jul 16 22:43:17.910819 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:17.910799    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3be5072-ee7d-4cd1-80e6-79eea0ae843b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3be5072-ee7d-4cd1-80e6-79eea0ae843b" (UID: "c3be5072-ee7d-4cd1-80e6-79eea0ae843b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:18.011791 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:18.011751    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3be5072-ee7d-4cd1-80e6-79eea0ae843b-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:43:18.621844 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:18.621815    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"
Jul 16 22:43:18.622045 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:18.621823    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz" event={"ID":"c3be5072-ee7d-4cd1-80e6-79eea0ae843b","Type":"ContainerDied","Data":"2db2566846e27a74e1675ff863ab1b1f5daf9847e1290dcb4eef021cbf159e58"}
Jul 16 22:43:18.622045 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:18.621950    2562 scope.go:117] "RemoveContainer" containerID="66f3acce2426af4a83d6924dcb78cecc0e9635d95fac05e72518039cc597a511"
Jul 16 22:43:18.647929 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:18.647900    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"]
Jul 16 22:43:18.655862 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:18.655839    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-zqbfz"]
Jul 16 22:43:19.751202 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:19.751170    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" path="/var/lib/kubelet/pods/c3be5072-ee7d-4cd1-80e6-79eea0ae843b/volumes"
Jul 16 22:43:23.671626 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671589    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"]
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671945    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671956    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671967    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671972    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671985    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.671991    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672005    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672010    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672012 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672016    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672286 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672021    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672286 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672073    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672286 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672080    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.672286 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672085    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672286 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672091    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:23.672286 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.672101    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:23.676524 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.676508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:23.679140 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.679121    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:43:23.682763 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.682744    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"]
Jul 16 22:43:23.762288 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.762258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6eda43-1a3e-44c9-b16e-2daaf436150e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-zt9nl\" (UID: \"7a6eda43-1a3e-44c9-b16e-2daaf436150e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:23.862751 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.862714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6eda43-1a3e-44c9-b16e-2daaf436150e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-zt9nl\" (UID: \"7a6eda43-1a3e-44c9-b16e-2daaf436150e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:23.863353 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.863334    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6eda43-1a3e-44c9-b16e-2daaf436150e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-zt9nl\" (UID: \"7a6eda43-1a3e-44c9-b16e-2daaf436150e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:23.988486 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:23.988389    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:24.149042 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:24.149016    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"]
Jul 16 22:43:24.152138 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:43:24.152112    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6eda43_1a3e_44c9_b16e_2daaf436150e.slice/crio-3fca4ad70172353f31bd43cc356c372fd27849850571a1003cd11a07333bb384 WatchSource:0}: Error finding container 3fca4ad70172353f31bd43cc356c372fd27849850571a1003cd11a07333bb384: Status 404 returned error can't find the container with id 3fca4ad70172353f31bd43cc356c372fd27849850571a1003cd11a07333bb384
Jul 16 22:43:24.655717 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:24.655679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" event={"ID":"7a6eda43-1a3e-44c9-b16e-2daaf436150e","Type":"ContainerStarted","Data":"2848cba93630809be31c2c0113c3fdc73543bb0fa2f3f55c447d4a8a1c3c0278"}
Jul 16 22:43:24.655911 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:24.655743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" event={"ID":"7a6eda43-1a3e-44c9-b16e-2daaf436150e","Type":"ContainerStarted","Data":"3fca4ad70172353f31bd43cc356c372fd27849850571a1003cd11a07333bb384"}
Jul 16 22:43:24.672265 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:24.672217    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" podStartSLOduration=1.672200468 podStartE2EDuration="1.672200468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:24.671371126 +0000 UTC m=+3387.453268975" watchObservedRunningTime="2026-07-16 22:43:24.672200468 +0000 UTC m=+3387.454098315"
Jul 16 22:43:25.662766 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:25.662731    2562 generic.go:356] "Generic (PLEG): container finished" podID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerID="2848cba93630809be31c2c0113c3fdc73543bb0fa2f3f55c447d4a8a1c3c0278" exitCode=6
Jul 16 22:43:25.662938 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:25.662813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" event={"ID":"7a6eda43-1a3e-44c9-b16e-2daaf436150e","Type":"ContainerDied","Data":"2848cba93630809be31c2c0113c3fdc73543bb0fa2f3f55c447d4a8a1c3c0278"}
Jul 16 22:43:25.663038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:25.662988    2562 scope.go:117] "RemoveContainer" containerID="2848cba93630809be31c2c0113c3fdc73543bb0fa2f3f55c447d4a8a1c3c0278"
Jul 16 22:43:26.669008 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:26.668970    2562 generic.go:356] "Generic (PLEG): container finished" podID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerID="12950a57aab369ac81966046d937b939f203f9e0b8d971e166a8c95dac1d42e9" exitCode=6
Jul 16 22:43:26.669495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:26.669059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" event={"ID":"7a6eda43-1a3e-44c9-b16e-2daaf436150e","Type":"ContainerDied","Data":"12950a57aab369ac81966046d937b939f203f9e0b8d971e166a8c95dac1d42e9"}
Jul 16 22:43:26.669495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:26.669130    2562 scope.go:117] "RemoveContainer" containerID="2848cba93630809be31c2c0113c3fdc73543bb0fa2f3f55c447d4a8a1c3c0278"
Jul 16 22:43:26.669495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:26.669295    2562 scope.go:117] "RemoveContainer" containerID="12950a57aab369ac81966046d937b939f203f9e0b8d971e166a8c95dac1d42e9"
Jul 16 22:43:26.669669 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:26.669508    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-zt9nl_odh-ai-gateway-infra(7a6eda43-1a3e-44c9-b16e-2daaf436150e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e"
Jul 16 22:43:27.675534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:27.675504    2562 scope.go:117] "RemoveContainer" containerID="12950a57aab369ac81966046d937b939f203f9e0b8d971e166a8c95dac1d42e9"
Jul 16 22:43:27.675981 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:27.675736    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-zt9nl_odh-ai-gateway-infra(7a6eda43-1a3e-44c9-b16e-2daaf436150e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e"
Jul 16 22:43:39.745541 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:39.745494    2562 scope.go:117] "RemoveContainer" containerID="12950a57aab369ac81966046d937b939f203f9e0b8d971e166a8c95dac1d42e9"
Jul 16 22:43:40.737818 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:40.737780    2562 generic.go:356] "Generic (PLEG): container finished" podID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerID="56d686dfad080ec228861c591f161da73562a03bcb721c1b7f03683c44d8f361" exitCode=6
Jul 16 22:43:40.738017 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:40.737840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" event={"ID":"7a6eda43-1a3e-44c9-b16e-2daaf436150e","Type":"ContainerDied","Data":"56d686dfad080ec228861c591f161da73562a03bcb721c1b7f03683c44d8f361"}
Jul 16 22:43:40.738017 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:40.737904    2562 scope.go:117] "RemoveContainer" containerID="12950a57aab369ac81966046d937b939f203f9e0b8d971e166a8c95dac1d42e9"
Jul 16 22:43:40.738232 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:40.738214    2562 scope.go:117] "RemoveContainer" containerID="56d686dfad080ec228861c591f161da73562a03bcb721c1b7f03683c44d8f361"
Jul 16 22:43:40.738451 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:40.738426    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-zt9nl_odh-ai-gateway-infra(7a6eda43-1a3e-44c9-b16e-2daaf436150e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e"
Jul 16 22:43:41.767977 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:41.767946    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"]
Jul 16 22:43:41.902991 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:41.902970    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:41.930836 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:41.930803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6eda43-1a3e-44c9-b16e-2daaf436150e-maas-api-service-ca\") pod \"7a6eda43-1a3e-44c9-b16e-2daaf436150e\" (UID: \"7a6eda43-1a3e-44c9-b16e-2daaf436150e\") "
Jul 16 22:43:41.931151 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:41.931127    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6eda43-1a3e-44c9-b16e-2daaf436150e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a6eda43-1a3e-44c9-b16e-2daaf436150e" (UID: "7a6eda43-1a3e-44c9-b16e-2daaf436150e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:42.031684 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:42.031602    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6eda43-1a3e-44c9-b16e-2daaf436150e-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:43:42.752747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:42.752715    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"
Jul 16 22:43:42.752747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:42.752725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl" event={"ID":"7a6eda43-1a3e-44c9-b16e-2daaf436150e","Type":"ContainerDied","Data":"3fca4ad70172353f31bd43cc356c372fd27849850571a1003cd11a07333bb384"}
Jul 16 22:43:42.753041 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:42.752781    2562 scope.go:117] "RemoveContainer" containerID="56d686dfad080ec228861c591f161da73562a03bcb721c1b7f03683c44d8f361"
Jul 16 22:43:42.786451 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:42.786425    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"]
Jul 16 22:43:42.794470 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:42.794447    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zt9nl"]
Jul 16 22:43:43.752195 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:43.752164    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" path="/var/lib/kubelet/pods/7a6eda43-1a3e-44c9-b16e-2daaf436150e/volumes"
Jul 16 22:43:46.094997 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.094964    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"]
Jul 16 22:43:46.095394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095311    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:43:46.095394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095322    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:43:46.095394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095329    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:43:46.095394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095335    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:43:46.095394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095348    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:46.095394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095354    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3be5072-ee7d-4cd1-80e6-79eea0ae843b" containerName="revoke-keys"
Jul 16 22:43:46.095613 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095420    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:43:46.095613 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095428    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:43:46.095613 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.095435    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb4a619-f791-40bf-bf73-61823f55cbe7" containerName="revoke-keys"
Jul 16 22:43:46.098649 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.098629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:43:46.102455 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.102435    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:43:46.109160 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.109126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"]
Jul 16 22:43:46.166103 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.166070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b34b1-44db-4db8-aea3-194c6ef6418a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt\" (UID: \"980b34b1-44db-4db8-aea3-194c6ef6418a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:43:46.266517 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.266481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b34b1-44db-4db8-aea3-194c6ef6418a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt\" (UID: \"980b34b1-44db-4db8-aea3-194c6ef6418a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:43:46.267130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.267111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b34b1-44db-4db8-aea3-194c6ef6418a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt\" (UID: \"980b34b1-44db-4db8-aea3-194c6ef6418a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:43:46.409184 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.409159    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:43:46.551747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.551716    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"]
Jul 16 22:43:46.553276 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:43:46.553249    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980b34b1_44db_4db8_aea3_194c6ef6418a.slice/crio-9776f38dcb0e1ce4d689818bbdbce4393031415dc778911a3f94db416ad48f06 WatchSource:0}: Error finding container 9776f38dcb0e1ce4d689818bbdbce4393031415dc778911a3f94db416ad48f06: Status 404 returned error can't find the container with id 9776f38dcb0e1ce4d689818bbdbce4393031415dc778911a3f94db416ad48f06
Jul 16 22:43:46.773500 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:46.773463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" event={"ID":"980b34b1-44db-4db8-aea3-194c6ef6418a","Type":"ContainerStarted","Data":"9776f38dcb0e1ce4d689818bbdbce4393031415dc778911a3f94db416ad48f06"}
Jul 16 22:43:47.779429 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:47.779396    2562 generic.go:356] "Generic (PLEG): container finished" podID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerID="d738bd2768afa9f6c3481f4ab586fe4f57eb0adee533482243dd4804422df6f3" exitCode=6
Jul 16 22:43:47.779896 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:47.779455    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" event={"ID":"980b34b1-44db-4db8-aea3-194c6ef6418a","Type":"ContainerDied","Data":"d738bd2768afa9f6c3481f4ab586fe4f57eb0adee533482243dd4804422df6f3"}
Jul 16 22:43:47.779896 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:47.779656    2562 scope.go:117] "RemoveContainer" containerID="d738bd2768afa9f6c3481f4ab586fe4f57eb0adee533482243dd4804422df6f3"
Jul 16 22:43:48.785635 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:48.785528    2562 generic.go:356] "Generic (PLEG): container finished" podID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerID="b9b49e2f50a5d792d4ccfc885720269a0fdff71c95f76eece6ae0af095fb0a97" exitCode=6
Jul 16 22:43:48.785635 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:48.785616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" event={"ID":"980b34b1-44db-4db8-aea3-194c6ef6418a","Type":"ContainerDied","Data":"b9b49e2f50a5d792d4ccfc885720269a0fdff71c95f76eece6ae0af095fb0a97"}
Jul 16 22:43:48.786143 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:48.785684    2562 scope.go:117] "RemoveContainer" containerID="d738bd2768afa9f6c3481f4ab586fe4f57eb0adee533482243dd4804422df6f3"
Jul 16 22:43:48.786143 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:48.785900    2562 scope.go:117] "RemoveContainer" containerID="b9b49e2f50a5d792d4ccfc885720269a0fdff71c95f76eece6ae0af095fb0a97"
Jul 16 22:43:48.786249 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:48.786134    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt_odh-ai-gateway-infra(980b34b1-44db-4db8-aea3-194c6ef6418a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a"
Jul 16 22:43:49.792069 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:49.792044    2562 scope.go:117] "RemoveContainer" containerID="b9b49e2f50a5d792d4ccfc885720269a0fdff71c95f76eece6ae0af095fb0a97"
Jul 16 22:43:49.792459 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:43:49.792219    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt_odh-ai-gateway-infra(980b34b1-44db-4db8-aea3-194c6ef6418a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a"
Jul 16 22:43:59.674846 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:43:59.674811    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:05.745172 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:05.745133    2562 scope.go:117] "RemoveContainer" containerID="b9b49e2f50a5d792d4ccfc885720269a0fdff71c95f76eece6ae0af095fb0a97"
Jul 16 22:44:06.877715 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:06.877625    2562 generic.go:356] "Generic (PLEG): container finished" podID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerID="74dcc3ebe5b75787f1bdbf565fb065b2dbd90b302c3bed96ef427b3c9894de06" exitCode=6
Jul 16 22:44:06.877715 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:06.877694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" event={"ID":"980b34b1-44db-4db8-aea3-194c6ef6418a","Type":"ContainerDied","Data":"74dcc3ebe5b75787f1bdbf565fb065b2dbd90b302c3bed96ef427b3c9894de06"}
Jul 16 22:44:06.878180 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:06.877760    2562 scope.go:117] "RemoveContainer" containerID="b9b49e2f50a5d792d4ccfc885720269a0fdff71c95f76eece6ae0af095fb0a97"
Jul 16 22:44:06.878180 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:06.878081    2562 scope.go:117] "RemoveContainer" containerID="74dcc3ebe5b75787f1bdbf565fb065b2dbd90b302c3bed96ef427b3c9894de06"
Jul 16 22:44:06.878334 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:06.878312    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt_odh-ai-gateway-infra(980b34b1-44db-4db8-aea3-194c6ef6418a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a"
Jul 16 22:44:06.970135 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:06.970110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:07.910785 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:07.910755    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"]
Jul 16 22:44:08.046135 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.046101    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:44:08.159465 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.159433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b34b1-44db-4db8-aea3-194c6ef6418a-maas-api-service-ca\") pod \"980b34b1-44db-4db8-aea3-194c6ef6418a\" (UID: \"980b34b1-44db-4db8-aea3-194c6ef6418a\") "
Jul 16 22:44:08.159859 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.159832    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980b34b1-44db-4db8-aea3-194c6ef6418a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "980b34b1-44db-4db8-aea3-194c6ef6418a" (UID: "980b34b1-44db-4db8-aea3-194c6ef6418a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:08.260838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.260756    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b34b1-44db-4db8-aea3-194c6ef6418a-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:08.890390 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.890353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt" event={"ID":"980b34b1-44db-4db8-aea3-194c6ef6418a","Type":"ContainerDied","Data":"9776f38dcb0e1ce4d689818bbdbce4393031415dc778911a3f94db416ad48f06"}
Jul 16 22:44:08.890604 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.890396    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"
Jul 16 22:44:08.890604 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.890417    2562 scope.go:117] "RemoveContainer" containerID="74dcc3ebe5b75787f1bdbf565fb065b2dbd90b302c3bed96ef427b3c9894de06"
Jul 16 22:44:08.918087 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.918061    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"]
Jul 16 22:44:08.923576 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:08.923531    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-dhcxt"]
Jul 16 22:44:09.751197 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:09.751166    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" path="/var/lib/kubelet/pods/980b34b1-44db-4db8-aea3-194c6ef6418a/volumes"
Jul 16 22:44:13.736828 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.736792    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"]
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737171    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737190    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737210    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737216    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737228    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737233    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737239    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737244 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737244    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737489 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737306    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6eda43-1a3e-44c9-b16e-2daaf436150e" containerName="revoke-keys"
Jul 16 22:44:13.737489 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737315    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.737489 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.737322    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:13.741759 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.741741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:13.745313 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.745286    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:44:13.763270 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.763236    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"]
Jul 16 22:44:13.811895 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.811865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322fe50a-94b9-4464-951e-b961f091c75d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-vxs4k\" (UID: \"322fe50a-94b9-4464-951e-b961f091c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:13.912471 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.912436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322fe50a-94b9-4464-951e-b961f091c75d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-vxs4k\" (UID: \"322fe50a-94b9-4464-951e-b961f091c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:13.913091 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:13.913069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322fe50a-94b9-4464-951e-b961f091c75d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-vxs4k\" (UID: \"322fe50a-94b9-4464-951e-b961f091c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:14.052723 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:14.052629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:14.209096 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:14.209072    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"]
Jul 16 22:44:14.211591 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:44:14.211566    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322fe50a_94b9_4464_951e_b961f091c75d.slice/crio-49d38b31fc318f3419803be117fff3db4491b9268d5ea257b42c8817e3166813 WatchSource:0}: Error finding container 49d38b31fc318f3419803be117fff3db4491b9268d5ea257b42c8817e3166813: Status 404 returned error can't find the container with id 49d38b31fc318f3419803be117fff3db4491b9268d5ea257b42c8817e3166813
Jul 16 22:44:14.920829 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:14.920798    2562 generic.go:356] "Generic (PLEG): container finished" podID="322fe50a-94b9-4464-951e-b961f091c75d" containerID="14d824fc3b44c600b6a99cb3744ae1ed153469110c305619e19151582f27dde2" exitCode=6
Jul 16 22:44:14.921301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:14.920882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" event={"ID":"322fe50a-94b9-4464-951e-b961f091c75d","Type":"ContainerDied","Data":"14d824fc3b44c600b6a99cb3744ae1ed153469110c305619e19151582f27dde2"}
Jul 16 22:44:14.921301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:14.920938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" event={"ID":"322fe50a-94b9-4464-951e-b961f091c75d","Type":"ContainerStarted","Data":"49d38b31fc318f3419803be117fff3db4491b9268d5ea257b42c8817e3166813"}
Jul 16 22:44:14.921301 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:14.921029    2562 scope.go:117] "RemoveContainer" containerID="14d824fc3b44c600b6a99cb3744ae1ed153469110c305619e19151582f27dde2"
Jul 16 22:44:15.927789 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:15.927755    2562 generic.go:356] "Generic (PLEG): container finished" podID="322fe50a-94b9-4464-951e-b961f091c75d" containerID="87ce06d2d7bfd3c9333213a15f21505e57c5c0307e814a42d20405be6a0ea083" exitCode=6
Jul 16 22:44:15.928250 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:15.927793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" event={"ID":"322fe50a-94b9-4464-951e-b961f091c75d","Type":"ContainerDied","Data":"87ce06d2d7bfd3c9333213a15f21505e57c5c0307e814a42d20405be6a0ea083"}
Jul 16 22:44:15.928250 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:15.927867    2562 scope.go:117] "RemoveContainer" containerID="14d824fc3b44c600b6a99cb3744ae1ed153469110c305619e19151582f27dde2"
Jul 16 22:44:15.928250 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:15.928042    2562 scope.go:117] "RemoveContainer" containerID="87ce06d2d7bfd3c9333213a15f21505e57c5c0307e814a42d20405be6a0ea083"
Jul 16 22:44:15.928250 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:15.928228    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-vxs4k_odh-ai-gateway-infra(322fe50a-94b9-4464-951e-b961f091c75d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" podUID="322fe50a-94b9-4464-951e-b961f091c75d"
Jul 16 22:44:16.934993 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:16.934968    2562 scope.go:117] "RemoveContainer" containerID="87ce06d2d7bfd3c9333213a15f21505e57c5c0307e814a42d20405be6a0ea083"
Jul 16 22:44:16.935380 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:16.935178    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-vxs4k_odh-ai-gateway-infra(322fe50a-94b9-4464-951e-b961f091c75d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" podUID="322fe50a-94b9-4464-951e-b961f091c75d"
Jul 16 22:44:25.407395 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.407357    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"]
Jul 16 22:44:25.407857 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.407842    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="980b34b1-44db-4db8-aea3-194c6ef6418a" containerName="revoke-keys"
Jul 16 22:44:25.412278 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.412261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:25.438331 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.438306    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"]
Jul 16 22:44:25.514871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.514836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6135d2f4-3813-48e5-92de-8a9c7a74e1aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt\" (UID: \"6135d2f4-3813-48e5-92de-8a9c7a74e1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:25.615688 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.615650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6135d2f4-3813-48e5-92de-8a9c7a74e1aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt\" (UID: \"6135d2f4-3813-48e5-92de-8a9c7a74e1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:25.616306 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.616280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6135d2f4-3813-48e5-92de-8a9c7a74e1aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt\" (UID: \"6135d2f4-3813-48e5-92de-8a9c7a74e1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:25.722454 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.722377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:25.856298 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.856276    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"]
Jul 16 22:44:25.858766 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:44:25.858741    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6135d2f4_3813_48e5_92de_8a9c7a74e1aa.slice/crio-ba43d212ec359993ad4d2359798da2519fabeeff6a3c4183cd5f6dbac995d0ef WatchSource:0}: Error finding container ba43d212ec359993ad4d2359798da2519fabeeff6a3c4183cd5f6dbac995d0ef: Status 404 returned error can't find the container with id ba43d212ec359993ad4d2359798da2519fabeeff6a3c4183cd5f6dbac995d0ef
Jul 16 22:44:25.979850 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:25.979759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" event={"ID":"6135d2f4-3813-48e5-92de-8a9c7a74e1aa","Type":"ContainerStarted","Data":"ba43d212ec359993ad4d2359798da2519fabeeff6a3c4183cd5f6dbac995d0ef"}
Jul 16 22:44:26.986251 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:26.986217    2562 generic.go:356] "Generic (PLEG): container finished" podID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerID="a45527d25bfa4964a40d316922d2652956f4ae5863817905ecba446d6a8efc34" exitCode=6
Jul 16 22:44:26.986680 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:26.986305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" event={"ID":"6135d2f4-3813-48e5-92de-8a9c7a74e1aa","Type":"ContainerDied","Data":"a45527d25bfa4964a40d316922d2652956f4ae5863817905ecba446d6a8efc34"}
Jul 16 22:44:26.986680 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:26.986437    2562 scope.go:117] "RemoveContainer" containerID="a45527d25bfa4964a40d316922d2652956f4ae5863817905ecba446d6a8efc34"
Jul 16 22:44:27.748051 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:27.748011    2562 scope.go:117] "RemoveContainer" containerID="87ce06d2d7bfd3c9333213a15f21505e57c5c0307e814a42d20405be6a0ea083"
Jul 16 22:44:27.993022 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:27.992992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" event={"ID":"322fe50a-94b9-4464-951e-b961f091c75d","Type":"ContainerStarted","Data":"c1f4d8b6e99f35bdd07001430e15ede1e274d6f9915d107807ca44c7516c7541"}
Jul 16 22:44:27.994750 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:27.994725    2562 generic.go:356] "Generic (PLEG): container finished" podID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerID="8c2686330f4134ad1c52037929ec074c060736e4851dfdefd0ab3219bba4a18d" exitCode=6
Jul 16 22:44:27.994847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:27.994809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" event={"ID":"6135d2f4-3813-48e5-92de-8a9c7a74e1aa","Type":"ContainerDied","Data":"8c2686330f4134ad1c52037929ec074c060736e4851dfdefd0ab3219bba4a18d"}
Jul 16 22:44:27.994905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:27.994868    2562 scope.go:117] "RemoveContainer" containerID="a45527d25bfa4964a40d316922d2652956f4ae5863817905ecba446d6a8efc34"
Jul 16 22:44:27.994959 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:27.994946    2562 scope.go:117] "RemoveContainer" containerID="8c2686330f4134ad1c52037929ec074c060736e4851dfdefd0ab3219bba4a18d"
Jul 16 22:44:27.995189 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:27.995169    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt_odh-ai-gateway-infra(6135d2f4-3813-48e5-92de-8a9c7a74e1aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa"
Jul 16 22:44:28.011532 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:28.011496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" podStartSLOduration=15.011483374 podStartE2EDuration="15.011483374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:28.01140111 +0000 UTC m=+3450.793298961" watchObservedRunningTime="2026-07-16 22:44:28.011483374 +0000 UTC m=+3450.793381221"
Jul 16 22:44:29.001617 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:29.001579    2562 generic.go:356] "Generic (PLEG): container finished" podID="322fe50a-94b9-4464-951e-b961f091c75d" containerID="c1f4d8b6e99f35bdd07001430e15ede1e274d6f9915d107807ca44c7516c7541" exitCode=6
Jul 16 22:44:29.002093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:29.001648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" event={"ID":"322fe50a-94b9-4464-951e-b961f091c75d","Type":"ContainerDied","Data":"c1f4d8b6e99f35bdd07001430e15ede1e274d6f9915d107807ca44c7516c7541"}
Jul 16 22:44:29.002093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:29.001715    2562 scope.go:117] "RemoveContainer" containerID="87ce06d2d7bfd3c9333213a15f21505e57c5c0307e814a42d20405be6a0ea083"
Jul 16 22:44:29.002093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:29.002034    2562 scope.go:117] "RemoveContainer" containerID="c1f4d8b6e99f35bdd07001430e15ede1e274d6f9915d107807ca44c7516c7541"
Jul 16 22:44:29.008336 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:29.002771    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-vxs4k_odh-ai-gateway-infra(322fe50a-94b9-4464-951e-b961f091c75d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" podUID="322fe50a-94b9-4464-951e-b961f091c75d"
Jul 16 22:44:29.010245 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:29.010218    2562 scope.go:117] "RemoveContainer" containerID="8c2686330f4134ad1c52037929ec074c060736e4851dfdefd0ab3219bba4a18d"
Jul 16 22:44:29.010440 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:29.010420    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt_odh-ai-gateway-infra(6135d2f4-3813-48e5-92de-8a9c7a74e1aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa"
Jul 16 22:44:29.023860 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:29.023837    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"]
Jul 16 22:44:30.152576 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:30.152536    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:30.258993 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:30.258965    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322fe50a-94b9-4464-951e-b961f091c75d-maas-api-service-ca\") pod \"322fe50a-94b9-4464-951e-b961f091c75d\" (UID: \"322fe50a-94b9-4464-951e-b961f091c75d\") "
Jul 16 22:44:30.259360 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:30.259340    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322fe50a-94b9-4464-951e-b961f091c75d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "322fe50a-94b9-4464-951e-b961f091c75d" (UID: "322fe50a-94b9-4464-951e-b961f091c75d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:30.359982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:30.359898    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322fe50a-94b9-4464-951e-b961f091c75d-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:31.021847 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:31.021819    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"
Jul 16 22:44:31.022035 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:31.021818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k" event={"ID":"322fe50a-94b9-4464-951e-b961f091c75d","Type":"ContainerDied","Data":"49d38b31fc318f3419803be117fff3db4491b9268d5ea257b42c8817e3166813"}
Jul 16 22:44:31.022035 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:31.021949    2562 scope.go:117] "RemoveContainer" containerID="c1f4d8b6e99f35bdd07001430e15ede1e274d6f9915d107807ca44c7516c7541"
Jul 16 22:44:31.046514 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:31.046481    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"]
Jul 16 22:44:31.052521 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:31.052499    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-vxs4k"]
Jul 16 22:44:31.750913 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:31.750872    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322fe50a-94b9-4464-951e-b961f091c75d" path="/var/lib/kubelet/pods/322fe50a-94b9-4464-951e-b961f091c75d/volumes"
Jul 16 22:44:36.237526 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:36.237477    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:37.030769 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:37.030733    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"]
Jul 16 22:44:37.058469 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:37.058419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:42.839318 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839284    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"]
Jul 16 22:44:42.839711 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839686    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839711 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839698    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839711 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839708    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839711 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839713    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839731    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839741    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839813    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839825    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.839839 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.839835    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="322fe50a-94b9-4464-951e-b961f091c75d" containerName="revoke-keys"
Jul 16 22:44:42.842926 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.842908    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.853778 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.853752    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"sub-test-model-fbed9b-kserve-self-signed-certs\""
Jul 16 22:44:42.853954 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.853854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"default-dockercfg-cjhlj\""
Jul 16 22:44:42.853954 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.853943    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"openshift-service-ca.crt\""
Jul 16 22:44:42.854149 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.853862    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"kube-root-ca.crt\""
Jul 16 22:44:42.863765 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.863725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-home\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.863914 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.863837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-model-cache\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.863914 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.863884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-tmp-dir\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.864043 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.863982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-dshm\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.864043 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.864011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27qj\" (UniqueName: \"kubernetes.io/projected/cfd33491-23e6-4e59-922b-dd99c527b119-kube-api-access-n27qj\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.864149 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.864043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd33491-23e6-4e59-922b-dd99c527b119-tls-certs\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.867593 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.867565    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"]
Jul 16 22:44:42.907513 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.907480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:42.964887 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.964853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-dshm\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.964887 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.964887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27qj\" (UniqueName: \"kubernetes.io/projected/cfd33491-23e6-4e59-922b-dd99c527b119-kube-api-access-n27qj\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.964911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd33491-23e6-4e59-922b-dd99c527b119-tls-certs\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.964942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-home\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.964988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-model-cache\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.965012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-tmp-dir\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.965353    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-model-cache\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965426 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.965401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-home\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.965473 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.965436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-tmp-dir\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.967323 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.967294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-dshm\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.967498 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.967484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd33491-23e6-4e59-922b-dd99c527b119-tls-certs\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:42.978573 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:42.978526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27qj\" (UniqueName: \"kubernetes.io/projected/cfd33491-23e6-4e59-922b-dd99c527b119-kube-api-access-n27qj\") pod \"sub-test-model-fbed9b-kserve-68fd648b99-mt455\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") " pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:43.153674 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:43.153639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:43.301866 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:43.301832    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"]
Jul 16 22:44:43.306832 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:44:43.306803    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfd33491_23e6_4e59_922b_dd99c527b119.slice/crio-04a8ef7681c57c60caa5f1cf05ad5a1f1143239d40391b9291de26b368e594e6 WatchSource:0}: Error finding container 04a8ef7681c57c60caa5f1cf05ad5a1f1143239d40391b9291de26b368e594e6: Status 404 returned error can't find the container with id 04a8ef7681c57c60caa5f1cf05ad5a1f1143239d40391b9291de26b368e594e6
Jul 16 22:44:43.745428 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:43.745395    2562 scope.go:117] "RemoveContainer" containerID="8c2686330f4134ad1c52037929ec074c060736e4851dfdefd0ab3219bba4a18d"
Jul 16 22:44:44.094199 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:44.094166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" event={"ID":"6135d2f4-3813-48e5-92de-8a9c7a74e1aa","Type":"ContainerStarted","Data":"e76b4e050cd111cdcc584004784d7a3289f4bcb572970a3a6049572d2f0d719c"}
Jul 16 22:44:44.095838 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:44.095815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455" event={"ID":"cfd33491-23e6-4e59-922b-dd99c527b119","Type":"ContainerStarted","Data":"6ccf4e7c91a2c2adcc50c00c3d9e0466681dbd63a8b8b95d399606d9e1a4d8ba"}
Jul 16 22:44:44.095926 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:44.095850    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455" event={"ID":"cfd33491-23e6-4e59-922b-dd99c527b119","Type":"ContainerStarted","Data":"04a8ef7681c57c60caa5f1cf05ad5a1f1143239d40391b9291de26b368e594e6"}
Jul 16 22:44:44.123223 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:44.123180    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" podStartSLOduration=19.123164819 podStartE2EDuration="19.123164819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:44.120818346 +0000 UTC m=+3466.902716206" watchObservedRunningTime="2026-07-16 22:44:44.123164819 +0000 UTC m=+3466.905062668"
Jul 16 22:44:44.152705 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:44.152665    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455" podStartSLOduration=2.152651053 podStartE2EDuration="2.152651053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:44.151126329 +0000 UTC m=+3466.933024191" watchObservedRunningTime="2026-07-16 22:44:44.152651053 +0000 UTC m=+3466.934548901"
Jul 16 22:44:45.103065 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:45.103034    2562 generic.go:356] "Generic (PLEG): container finished" podID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerID="e76b4e050cd111cdcc584004784d7a3289f4bcb572970a3a6049572d2f0d719c" exitCode=6
Jul 16 22:44:45.103598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:45.103094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" event={"ID":"6135d2f4-3813-48e5-92de-8a9c7a74e1aa","Type":"ContainerDied","Data":"e76b4e050cd111cdcc584004784d7a3289f4bcb572970a3a6049572d2f0d719c"}
Jul 16 22:44:45.103598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:45.103159    2562 scope.go:117] "RemoveContainer" containerID="8c2686330f4134ad1c52037929ec074c060736e4851dfdefd0ab3219bba4a18d"
Jul 16 22:44:45.103598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:45.103531    2562 scope.go:117] "RemoveContainer" containerID="e76b4e050cd111cdcc584004784d7a3289f4bcb572970a3a6049572d2f0d719c"
Jul 16 22:44:45.103784 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:45.103721    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt_odh-ai-gateway-infra(6135d2f4-3813-48e5-92de-8a9c7a74e1aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa"
Jul 16 22:44:45.132614 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:45.132580    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"]
Jul 16 22:44:46.245018 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:46.244989    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:46.296641 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:46.296614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6135d2f4-3813-48e5-92de-8a9c7a74e1aa-maas-api-service-ca\") pod \"6135d2f4-3813-48e5-92de-8a9c7a74e1aa\" (UID: \"6135d2f4-3813-48e5-92de-8a9c7a74e1aa\") "
Jul 16 22:44:46.296975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:46.296951    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6135d2f4-3813-48e5-92de-8a9c7a74e1aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6135d2f4-3813-48e5-92de-8a9c7a74e1aa" (UID: "6135d2f4-3813-48e5-92de-8a9c7a74e1aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:46.397462 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:46.397374    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6135d2f4-3813-48e5-92de-8a9c7a74e1aa-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:47.115790 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:47.115761    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"
Jul 16 22:44:47.115979 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:47.115803    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt" event={"ID":"6135d2f4-3813-48e5-92de-8a9c7a74e1aa","Type":"ContainerDied","Data":"ba43d212ec359993ad4d2359798da2519fabeeff6a3c4183cd5f6dbac995d0ef"}
Jul 16 22:44:47.115979 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:47.115877    2562 scope.go:117] "RemoveContainer" containerID="e76b4e050cd111cdcc584004784d7a3289f4bcb572970a3a6049572d2f0d719c"
Jul 16 22:44:47.152519 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:47.152484    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"]
Jul 16 22:44:47.159409 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:47.159383    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-jm6lt"]
Jul 16 22:44:47.751949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:47.751916    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" path="/var/lib/kubelet/pods/6135d2f4-3813-48e5-92de-8a9c7a74e1aa/volumes"
Jul 16 22:44:52.055652 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.055605    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth" podUID="776d6934-2835-41c5-8388-11b3757217ef" containerName="main" containerID="cri-o://7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437" gracePeriod=45
Jul 16 22:44:52.308848 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.308779    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:44:52.352280 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352236    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-home\") pod \"776d6934-2835-41c5-8388-11b3757217ef\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") "
Jul 16 22:44:52.352280 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352282    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-tmp-dir\") pod \"776d6934-2835-41c5-8388-11b3757217ef\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") "
Jul 16 22:44:52.352529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqwvs\" (UniqueName: \"kubernetes.io/projected/776d6934-2835-41c5-8388-11b3757217ef-kube-api-access-xqwvs\") pod \"776d6934-2835-41c5-8388-11b3757217ef\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") "
Jul 16 22:44:52.352529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352448    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-model-cache\") pod \"776d6934-2835-41c5-8388-11b3757217ef\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") "
Jul 16 22:44:52.352529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776d6934-2835-41c5-8388-11b3757217ef-tls-certs\") pod \"776d6934-2835-41c5-8388-11b3757217ef\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") "
Jul 16 22:44:52.352529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352509    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-home" (OuterVolumeSpecName: "home") pod "776d6934-2835-41c5-8388-11b3757217ef" (UID: "776d6934-2835-41c5-8388-11b3757217ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:52.352529 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-dshm\") pod \"776d6934-2835-41c5-8388-11b3757217ef\" (UID: \"776d6934-2835-41c5-8388-11b3757217ef\") "
Jul 16 22:44:52.353050 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.352836    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-home\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:52.353050 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.353002    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "776d6934-2835-41c5-8388-11b3757217ef" (UID: "776d6934-2835-41c5-8388-11b3757217ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:52.353325 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.353302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "776d6934-2835-41c5-8388-11b3757217ef" (UID: "776d6934-2835-41c5-8388-11b3757217ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:52.355538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.355503    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776d6934-2835-41c5-8388-11b3757217ef-kube-api-access-xqwvs" (OuterVolumeSpecName: "kube-api-access-xqwvs") pod "776d6934-2835-41c5-8388-11b3757217ef" (UID: "776d6934-2835-41c5-8388-11b3757217ef"). InnerVolumeSpecName "kube-api-access-xqwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:44:52.355685 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.355603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776d6934-2835-41c5-8388-11b3757217ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "776d6934-2835-41c5-8388-11b3757217ef" (UID: "776d6934-2835-41c5-8388-11b3757217ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:44:52.355685 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.355611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-dshm" (OuterVolumeSpecName: "dshm") pod "776d6934-2835-41c5-8388-11b3757217ef" (UID: "776d6934-2835-41c5-8388-11b3757217ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:52.453830 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.453788    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-tmp-dir\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:52.453830 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.453826    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwvs\" (UniqueName: \"kubernetes.io/projected/776d6934-2835-41c5-8388-11b3757217ef-kube-api-access-xqwvs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:52.454054 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.453841    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-model-cache\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:52.454054 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.453854    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776d6934-2835-41c5-8388-11b3757217ef-tls-certs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:52.454054 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:52.453865    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776d6934-2835-41c5-8388-11b3757217ef-dshm\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:44:53.144776 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.144740    2562 generic.go:356] "Generic (PLEG): container finished" podID="776d6934-2835-41c5-8388-11b3757217ef" containerID="7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437" exitCode=0
Jul 16 22:44:53.145348 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.144786    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"
Jul 16 22:44:53.145348 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.144831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth" event={"ID":"776d6934-2835-41c5-8388-11b3757217ef","Type":"ContainerDied","Data":"7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437"}
Jul 16 22:44:53.145348 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.144877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth" event={"ID":"776d6934-2835-41c5-8388-11b3757217ef","Type":"ContainerDied","Data":"b937f492d2e8c38915f31b6ac95f4e97f4d79d7cf88d3357d031cb1af65ccc91"}
Jul 16 22:44:53.145348 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.144898    2562 scope.go:117] "RemoveContainer" containerID="7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437"
Jul 16 22:44:53.154027 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.153745    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:53.154027 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.153784    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:53.155089 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.155063    2562 scope.go:117] "RemoveContainer" containerID="7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437"
Jul 16 22:44:53.155356 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:44:53.155332    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437\": container with ID starting with 7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437 not found: ID does not exist" containerID="7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437"
Jul 16 22:44:53.155445 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.155367    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437"} err="failed to get container status \"7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437\": rpc error: code = NotFound desc = could not find container \"7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437\": container with ID starting with 7423a31d4c0c78bd6f3a2611a7d4b51b2b08682f5d99a919bc5b4fd3904d9437 not found: ID does not exist"
Jul 16 22:44:53.167265 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.167209    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:44:53.169297 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.169251    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"]
Jul 16 22:44:53.171220 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.171199    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/auth-test-model-106c85-kserve-676b84c6c5-g7vth"]
Jul 16 22:44:53.751464 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:53.751424    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776d6934-2835-41c5-8388-11b3757217ef" path="/var/lib/kubelet/pods/776d6934-2835-41c5-8388-11b3757217ef/volumes"
Jul 16 22:44:54.162981 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:44:54.162952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:45:00.190702 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.190670    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"]
Jul 16 22:45:00.191104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191011    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:00.191104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191022    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:00.191104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191033    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776d6934-2835-41c5-8388-11b3757217ef" containerName="main"
Jul 16 22:45:00.191104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191039    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="776d6934-2835-41c5-8388-11b3757217ef" containerName="main"
Jul 16 22:45:00.191104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191051    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:00.191104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:00.191293 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191116    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:00.191293 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191127    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:00.191293 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.191134    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="776d6934-2835-41c5-8388-11b3757217ef" containerName="main"
Jul 16 22:45:00.195619 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.195601    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"
Jul 16 22:45:00.207097 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.207073    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:45:00.215461 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.215438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"
Jul 16 22:45:00.223993 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.223940    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"]
Jul 16 22:45:00.374489 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:00.374405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"]
Jul 16 22:45:00.376924 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:45:00.376892    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f082cb_f2c5_4f15_ad42_2e913bc61eae.slice/crio-101bfd9a055ffb9b13cb2f3866f2de38f32dab03f6b7c3457c83186c5722b601 WatchSource:0}: Error finding container 101bfd9a055ffb9b13cb2f3866f2de38f32dab03f6b7c3457c83186c5722b601: Status 404 returned error can't find the container with id 101bfd9a055ffb9b13cb2f3866f2de38f32dab03f6b7c3457c83186c5722b601
Jul 16 22:45:01.185808 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:01.185774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt" event={"ID":"76f082cb-f2c5-4f15-ad42-2e913bc61eae","Type":"ContainerStarted","Data":"ae0790e51b4c4b516691e01f449d63ec5b5b4bf489582be34f68fa18282edb09"}
Jul 16 22:45:01.185994 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:01.185824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt" event={"ID":"76f082cb-f2c5-4f15-ad42-2e913bc61eae","Type":"ContainerStarted","Data":"101bfd9a055ffb9b13cb2f3866f2de38f32dab03f6b7c3457c83186c5722b601"}
Jul 16 22:45:01.214822 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:01.214762    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt" podStartSLOduration=1.214746248 podStartE2EDuration="1.214746248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:01.213189685 +0000 UTC m=+3483.995087533" watchObservedRunningTime="2026-07-16 22:45:01.214746248 +0000 UTC m=+3483.996644096"
Jul 16 22:45:02.191989 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:02.191953    2562 generic.go:356] "Generic (PLEG): container finished" podID="76f082cb-f2c5-4f15-ad42-2e913bc61eae" containerID="ae0790e51b4c4b516691e01f449d63ec5b5b4bf489582be34f68fa18282edb09" exitCode=0
Jul 16 22:45:02.192186 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:02.192008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt" event={"ID":"76f082cb-f2c5-4f15-ad42-2e913bc61eae","Type":"ContainerDied","Data":"ae0790e51b4c4b516691e01f449d63ec5b5b4bf489582be34f68fa18282edb09"}
Jul 16 22:45:03.333992 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:03.333969    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"
Jul 16 22:45:04.203315 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:04.203286    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt"
Jul 16 22:45:04.203315 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:04.203301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-7dhpt" event={"ID":"76f082cb-f2c5-4f15-ad42-2e913bc61eae","Type":"ContainerDied","Data":"101bfd9a055ffb9b13cb2f3866f2de38f32dab03f6b7c3457c83186c5722b601"}
Jul 16 22:45:04.203539 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:04.203349    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="101bfd9a055ffb9b13cb2f3866f2de38f32dab03f6b7c3457c83186c5722b601"
Jul 16 22:45:04.401005 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:04.400965    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"]
Jul 16 22:45:04.406243 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:04.406219    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-zp5t7"]
Jul 16 22:45:05.751606 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:05.751574    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc81bf62-2a28-45cf-b7c6-7821758b4421" path="/var/lib/kubelet/pods/dc81bf62-2a28-45cf-b7c6-7821758b4421/volumes"
Jul 16 22:45:07.133955 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:07.133924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:07.230571 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:07.230519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:13.236644 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:13.236606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:37.874190 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874146    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"]
Jul 16 22:45:37.876933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874676    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:37.876933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874692    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:37.876933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874708    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f082cb-f2c5-4f15-ad42-2e913bc61eae" containerName="cleanup"
Jul 16 22:45:37.876933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874714    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f082cb-f2c5-4f15-ad42-2e913bc61eae" containerName="cleanup"
Jul 16 22:45:37.876933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874782    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f082cb-f2c5-4f15-ad42-2e913bc61eae" containerName="cleanup"
Jul 16 22:45:37.876933 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.874788    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6135d2f4-3813-48e5-92de-8a9c7a74e1aa" containerName="revoke-keys"
Jul 16 22:45:37.877890 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.877871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:37.880624 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.880601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:45:37.892588 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.888606    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"]
Jul 16 22:45:37.971310 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:37.971269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25af7d9b-187b-41aa-94da-f58ae1ffdbfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls\" (UID: \"25af7d9b-187b-41aa-94da-f58ae1ffdbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:38.072346 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:38.072311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25af7d9b-187b-41aa-94da-f58ae1ffdbfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls\" (UID: \"25af7d9b-187b-41aa-94da-f58ae1ffdbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:38.073014 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:38.072992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25af7d9b-187b-41aa-94da-f58ae1ffdbfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls\" (UID: \"25af7d9b-187b-41aa-94da-f58ae1ffdbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:38.195442 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:38.195405    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:38.331457 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:38.331428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"]
Jul 16 22:45:38.334007 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:45:38.333984    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25af7d9b_187b_41aa_94da_f58ae1ffdbfa.slice/crio-f8fccdda8389fb95caa5249f6a40ed99b85ef690596cb6a20a643d0750b643d5 WatchSource:0}: Error finding container f8fccdda8389fb95caa5249f6a40ed99b85ef690596cb6a20a643d0750b643d5: Status 404 returned error can't find the container with id f8fccdda8389fb95caa5249f6a40ed99b85ef690596cb6a20a643d0750b643d5
Jul 16 22:45:38.371509 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:38.371479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" event={"ID":"25af7d9b-187b-41aa-94da-f58ae1ffdbfa","Type":"ContainerStarted","Data":"f8fccdda8389fb95caa5249f6a40ed99b85ef690596cb6a20a643d0750b643d5"}
Jul 16 22:45:39.377973 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:39.377878    2562 generic.go:356] "Generic (PLEG): container finished" podID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerID="9d16c1964be42d57aecf4a1ee1f965d3713980c34bcbf8c334041ee485aea987" exitCode=6
Jul 16 22:45:39.377973 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:39.377957    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" event={"ID":"25af7d9b-187b-41aa-94da-f58ae1ffdbfa","Type":"ContainerDied","Data":"9d16c1964be42d57aecf4a1ee1f965d3713980c34bcbf8c334041ee485aea987"}
Jul 16 22:45:39.378375 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:39.378109    2562 scope.go:117] "RemoveContainer" containerID="9d16c1964be42d57aecf4a1ee1f965d3713980c34bcbf8c334041ee485aea987"
Jul 16 22:45:40.384504 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:40.384411    2562 generic.go:356] "Generic (PLEG): container finished" podID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerID="6d7668843e2f68bbede33621d950990d16890e211362edd626f2002ae2620c30" exitCode=6
Jul 16 22:45:40.384504 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:40.384471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" event={"ID":"25af7d9b-187b-41aa-94da-f58ae1ffdbfa","Type":"ContainerDied","Data":"6d7668843e2f68bbede33621d950990d16890e211362edd626f2002ae2620c30"}
Jul 16 22:45:40.385007 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:40.384516    2562 scope.go:117] "RemoveContainer" containerID="9d16c1964be42d57aecf4a1ee1f965d3713980c34bcbf8c334041ee485aea987"
Jul 16 22:45:40.385007 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:40.384721    2562 scope.go:117] "RemoveContainer" containerID="6d7668843e2f68bbede33621d950990d16890e211362edd626f2002ae2620c30"
Jul 16 22:45:40.385007 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:45:40.384932    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls_odh-ai-gateway-infra(25af7d9b-187b-41aa-94da-f58ae1ffdbfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa"
Jul 16 22:45:41.390755 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:41.390719    2562 scope.go:117] "RemoveContainer" containerID="6d7668843e2f68bbede33621d950990d16890e211362edd626f2002ae2620c30"
Jul 16 22:45:41.391406 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:45:41.390935    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls_odh-ai-gateway-infra(25af7d9b-187b-41aa-94da-f58ae1ffdbfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa"
Jul 16 22:45:46.227680 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.227640    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"]
Jul 16 22:45:46.231042 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.231024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:45:46.237852 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.237825    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"]
Jul 16 22:45:46.343575 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.343513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7\" (UID: \"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:45:46.444820 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.444781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7\" (UID: \"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:45:46.445410 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.445387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7\" (UID: \"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:45:46.542130 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.542045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:45:46.684926 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:46.684902    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"]
Jul 16 22:45:46.687153 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:45:46.687126    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28768b3b_8d00_4f9a_9d12_6e5d6fd455f1.slice/crio-840ff387a467d035cb6c2c2073875a47a80eea2f3d51fc51ecaec1d6ef79ea09 WatchSource:0}: Error finding container 840ff387a467d035cb6c2c2073875a47a80eea2f3d51fc51ecaec1d6ef79ea09: Status 404 returned error can't find the container with id 840ff387a467d035cb6c2c2073875a47a80eea2f3d51fc51ecaec1d6ef79ea09
Jul 16 22:45:47.421067 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:47.421007    2562 generic.go:356] "Generic (PLEG): container finished" podID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerID="f6aa9f196695234eb2b5294fb4b624290e5d6116a8693838252e3bd8b683f16d" exitCode=6
Jul 16 22:45:47.421067 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:47.421049    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" event={"ID":"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1","Type":"ContainerDied","Data":"f6aa9f196695234eb2b5294fb4b624290e5d6116a8693838252e3bd8b683f16d"}
Jul 16 22:45:47.421640 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:47.421099    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" event={"ID":"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1","Type":"ContainerStarted","Data":"840ff387a467d035cb6c2c2073875a47a80eea2f3d51fc51ecaec1d6ef79ea09"}
Jul 16 22:45:47.421640 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:47.421288    2562 scope.go:117] "RemoveContainer" containerID="f6aa9f196695234eb2b5294fb4b624290e5d6116a8693838252e3bd8b683f16d"
Jul 16 22:45:48.428367 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:48.428338    2562 generic.go:356] "Generic (PLEG): container finished" podID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerID="cc62be458711718e232a83619e625c2a7c4b65dbbbbeb6f66bb32e81de9d03e1" exitCode=6
Jul 16 22:45:48.428841 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:48.428373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" event={"ID":"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1","Type":"ContainerDied","Data":"cc62be458711718e232a83619e625c2a7c4b65dbbbbeb6f66bb32e81de9d03e1"}
Jul 16 22:45:48.428841 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:48.428413    2562 scope.go:117] "RemoveContainer" containerID="f6aa9f196695234eb2b5294fb4b624290e5d6116a8693838252e3bd8b683f16d"
Jul 16 22:45:48.428841 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:48.428651    2562 scope.go:117] "RemoveContainer" containerID="cc62be458711718e232a83619e625c2a7c4b65dbbbbeb6f66bb32e81de9d03e1"
Jul 16 22:45:48.428987 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:45:48.428876    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7_odh-ai-gateway-infra(28768b3b-8d00-4f9a-9d12-6e5d6fd455f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1"
Jul 16 22:45:49.434782 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:49.434755    2562 scope.go:117] "RemoveContainer" containerID="cc62be458711718e232a83619e625c2a7c4b65dbbbbeb6f66bb32e81de9d03e1"
Jul 16 22:45:49.435168 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:45:49.434990    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7_odh-ai-gateway-infra(28768b3b-8d00-4f9a-9d12-6e5d6fd455f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1"
Jul 16 22:45:53.745481 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:53.745439    2562 scope.go:117] "RemoveContainer" containerID="6d7668843e2f68bbede33621d950990d16890e211362edd626f2002ae2620c30"
Jul 16 22:45:54.460582 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:54.460529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" event={"ID":"25af7d9b-187b-41aa-94da-f58ae1ffdbfa","Type":"ContainerStarted","Data":"d1c35be4d6f475c5f551c7332afd120b24ef40485111b5253151ffb9668d4260"}
Jul 16 22:45:54.477083 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:54.477039    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" podStartSLOduration=17.477024729 podStartE2EDuration="17.477024729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:54.476014797 +0000 UTC m=+3537.257912647" watchObservedRunningTime="2026-07-16 22:45:54.477024729 +0000 UTC m=+3537.258922577"
Jul 16 22:45:55.467494 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:55.467458    2562 generic.go:356] "Generic (PLEG): container finished" podID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerID="d1c35be4d6f475c5f551c7332afd120b24ef40485111b5253151ffb9668d4260" exitCode=6
Jul 16 22:45:55.467897 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:55.467532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" event={"ID":"25af7d9b-187b-41aa-94da-f58ae1ffdbfa","Type":"ContainerDied","Data":"d1c35be4d6f475c5f551c7332afd120b24ef40485111b5253151ffb9668d4260"}
Jul 16 22:45:55.467897 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:55.467628    2562 scope.go:117] "RemoveContainer" containerID="6d7668843e2f68bbede33621d950990d16890e211362edd626f2002ae2620c30"
Jul 16 22:45:55.468021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:55.467945    2562 scope.go:117] "RemoveContainer" containerID="d1c35be4d6f475c5f551c7332afd120b24ef40485111b5253151ffb9668d4260"
Jul 16 22:45:55.468144 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:45:55.468124    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls_odh-ai-gateway-infra(25af7d9b-187b-41aa-94da-f58ae1ffdbfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa"
Jul 16 22:45:55.486081 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:55.486059    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"]
Jul 16 22:45:56.608861 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:56.608837    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:56.738651 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:56.738543    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25af7d9b-187b-41aa-94da-f58ae1ffdbfa-maas-api-service-ca\") pod \"25af7d9b-187b-41aa-94da-f58ae1ffdbfa\" (UID: \"25af7d9b-187b-41aa-94da-f58ae1ffdbfa\") "
Jul 16 22:45:56.738871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:56.738850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25af7d9b-187b-41aa-94da-f58ae1ffdbfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25af7d9b-187b-41aa-94da-f58ae1ffdbfa" (UID: "25af7d9b-187b-41aa-94da-f58ae1ffdbfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:56.839589 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:56.839528    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25af7d9b-187b-41aa-94da-f58ae1ffdbfa-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:45:57.480567 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:57.480510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls" event={"ID":"25af7d9b-187b-41aa-94da-f58ae1ffdbfa","Type":"ContainerDied","Data":"f8fccdda8389fb95caa5249f6a40ed99b85ef690596cb6a20a643d0750b643d5"}
Jul 16 22:45:57.480773 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:57.480526    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"
Jul 16 22:45:57.480773 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:57.480593    2562 scope.go:117] "RemoveContainer" containerID="d1c35be4d6f475c5f551c7332afd120b24ef40485111b5253151ffb9668d4260"
Jul 16 22:45:57.505877 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:57.505846    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"]
Jul 16 22:45:57.512228 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:57.512206    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ktkls"]
Jul 16 22:45:57.751829 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:57.751753    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" path="/var/lib/kubelet/pods/25af7d9b-187b-41aa-94da-f58ae1ffdbfa/volumes"
Jul 16 22:45:58.230397 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:45:58.230365    2562 scope.go:117] "RemoveContainer" containerID="eb1c040ef0a2ceea2ab68e3e68296b24eee8c9ecbf0b8deb654bcbe049356246"
Jul 16 22:46:00.746021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:00.745985    2562 scope.go:117] "RemoveContainer" containerID="cc62be458711718e232a83619e625c2a7c4b65dbbbbeb6f66bb32e81de9d03e1"
Jul 16 22:46:01.502444 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:01.502415    2562 generic.go:356] "Generic (PLEG): container finished" podID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerID="b6b7c2e07813525acfa9f450f3c312c2360e7267b134712aa22591260149a7f0" exitCode=6
Jul 16 22:46:01.502593 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:01.502479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" event={"ID":"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1","Type":"ContainerDied","Data":"b6b7c2e07813525acfa9f450f3c312c2360e7267b134712aa22591260149a7f0"}
Jul 16 22:46:01.502593 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:01.502528    2562 scope.go:117] "RemoveContainer" containerID="cc62be458711718e232a83619e625c2a7c4b65dbbbbeb6f66bb32e81de9d03e1"
Jul 16 22:46:01.502833 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:01.502813    2562 scope.go:117] "RemoveContainer" containerID="b6b7c2e07813525acfa9f450f3c312c2360e7267b134712aa22591260149a7f0"
Jul 16 22:46:01.503041 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:46:01.503016    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7_odh-ai-gateway-infra(28768b3b-8d00-4f9a-9d12-6e5d6fd455f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1"
Jul 16 22:46:02.532472 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:02.532440    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"]
Jul 16 22:46:02.669238 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:02.669208    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:46:02.695094 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:02.695065    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1-maas-api-service-ca\") pod \"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1\" (UID: \"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1\") "
Jul 16 22:46:02.695431 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:02.695408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" (UID: "28768b3b-8d00-4f9a-9d12-6e5d6fd455f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:02.795791 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:02.795705    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:03.514928 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:03.514895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7" event={"ID":"28768b3b-8d00-4f9a-9d12-6e5d6fd455f1","Type":"ContainerDied","Data":"840ff387a467d035cb6c2c2073875a47a80eea2f3d51fc51ecaec1d6ef79ea09"}
Jul 16 22:46:03.514928 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:03.514924    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"
Jul 16 22:46:03.515175 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:03.514954    2562 scope.go:117] "RemoveContainer" containerID="b6b7c2e07813525acfa9f450f3c312c2360e7267b134712aa22591260149a7f0"
Jul 16 22:46:03.537156 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:03.537127    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"]
Jul 16 22:46:03.538816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:03.538796    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xn2f7"]
Jul 16 22:46:03.750423 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:03.750389    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" path="/var/lib/kubelet/pods/28768b3b-8d00-4f9a-9d12-6e5d6fd455f1/volumes"
Jul 16 22:46:06.459799 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:06.459763    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"]
Jul 16 22:46:06.482186 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:06.482155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:07.310884 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:07.310825    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:15.040827 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:15.040790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:18.445361 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445320    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"]
Jul 16 22:46:18.445926 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445906    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:18.445975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445931    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:18.445975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445956    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:18.445975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445965    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:18.445975 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445974    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:18.446101 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445982    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:18.446101 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.445993    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:18.446101 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.446001    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:18.446101 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.446098    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:18.446222 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.446109    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:18.446222 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.446116    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:18.446222 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.446128    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:18.449677 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.449655    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.454022 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.453999    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"rate-test-model-fbed9b-kserve-self-signed-certs\""
Jul 16 22:46:18.462308 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.462284    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"]
Jul 16 22:46:18.481021 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.480999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:18.542602 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.542575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-home\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.542753 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.542633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-tmp-dir\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.542753 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.542705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7zg\" (UniqueName: \"kubernetes.io/projected/b5b312cc-a830-4334-b2ff-e3c533f0840e-kube-api-access-rr7zg\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.542753 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.542738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b312cc-a830-4334-b2ff-e3c533f0840e-tls-certs\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.542903 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.542803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-dshm\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.542903 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.542829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-model-cache\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644253 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-tmp-dir\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644425 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7zg\" (UniqueName: \"kubernetes.io/projected/b5b312cc-a830-4334-b2ff-e3c533f0840e-kube-api-access-rr7zg\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644425 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b312cc-a830-4334-b2ff-e3c533f0840e-tls-certs\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644425 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-dshm\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644425 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-model-cache\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644669 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-home\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644669 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-tmp-dir\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644831 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-home\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.644930 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.644910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-model-cache\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.646780 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.646763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-dshm\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.646950 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.646935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b312cc-a830-4334-b2ff-e3c533f0840e-tls-certs\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.652264 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.652238    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7zg\" (UniqueName: \"kubernetes.io/projected/b5b312cc-a830-4334-b2ff-e3c533f0840e-kube-api-access-rr7zg\") pod \"rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") " pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.759326 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.759240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:18.897164 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:18.897141    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"]
Jul 16 22:46:18.899261 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:46:18.899231    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5b312cc_a830_4334_b2ff_e3c533f0840e.slice/crio-e31438b40809ec4b59300a627e938f8cd0f0e11731173b7064f4d807ea1ace2e WatchSource:0}: Error finding container e31438b40809ec4b59300a627e938f8cd0f0e11731173b7064f4d807ea1ace2e: Status 404 returned error can't find the container with id e31438b40809ec4b59300a627e938f8cd0f0e11731173b7064f4d807ea1ace2e
Jul 16 22:46:19.596705 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:19.596671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5" event={"ID":"b5b312cc-a830-4334-b2ff-e3c533f0840e","Type":"ContainerStarted","Data":"e3778d1eedd0a08ef9d69727cc9fbc9e1a377f926dc04aa861f236c1324b816c"}
Jul 16 22:46:19.597119 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:19.596716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5" event={"ID":"b5b312cc-a830-4334-b2ff-e3c533f0840e","Type":"ContainerStarted","Data":"e31438b40809ec4b59300a627e938f8cd0f0e11731173b7064f4d807ea1ace2e"}
Jul 16 22:46:19.613938 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:19.613883    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5" podStartSLOduration=1.61386496 podStartE2EDuration="1.61386496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:19.613027329 +0000 UTC m=+3562.394925176" watchObservedRunningTime="2026-07-16 22:46:19.61386496 +0000 UTC m=+3562.395762811"
Jul 16 22:46:21.482566 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.482499    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455" podUID="cfd33491-23e6-4e59-922b-dd99c527b119" containerName="main" containerID="cri-o://6ccf4e7c91a2c2adcc50c00c3d9e0466681dbd63a8b8b95d399606d9e1a4d8ba" gracePeriod=45
Jul 16 22:46:21.609785 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.609756    2562 generic.go:356] "Generic (PLEG): container finished" podID="cfd33491-23e6-4e59-922b-dd99c527b119" containerID="6ccf4e7c91a2c2adcc50c00c3d9e0466681dbd63a8b8b95d399606d9e1a4d8ba" exitCode=0
Jul 16 22:46:21.609904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.609835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455" event={"ID":"cfd33491-23e6-4e59-922b-dd99c527b119","Type":"ContainerDied","Data":"6ccf4e7c91a2c2adcc50c00c3d9e0466681dbd63a8b8b95d399606d9e1a4d8ba"}
Jul 16 22:46:21.728691 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.728668    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:46:21.874648 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-dshm\") pod \"cfd33491-23e6-4e59-922b-dd99c527b119\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") "
Jul 16 22:46:21.874648 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-home\") pod \"cfd33491-23e6-4e59-922b-dd99c527b119\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") "
Jul 16 22:46:21.874648 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874626    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-model-cache\") pod \"cfd33491-23e6-4e59-922b-dd99c527b119\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") "
Jul 16 22:46:21.874648 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd33491-23e6-4e59-922b-dd99c527b119-tls-certs\") pod \"cfd33491-23e6-4e59-922b-dd99c527b119\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") "
Jul 16 22:46:21.874980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n27qj\" (UniqueName: \"kubernetes.io/projected/cfd33491-23e6-4e59-922b-dd99c527b119-kube-api-access-n27qj\") pod \"cfd33491-23e6-4e59-922b-dd99c527b119\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") "
Jul 16 22:46:21.874980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874763    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-tmp-dir\") pod \"cfd33491-23e6-4e59-922b-dd99c527b119\" (UID: \"cfd33491-23e6-4e59-922b-dd99c527b119\") "
Jul 16 22:46:21.874980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874852    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-home" (OuterVolumeSpecName: "home") pod "cfd33491-23e6-4e59-922b-dd99c527b119" (UID: "cfd33491-23e6-4e59-922b-dd99c527b119"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:21.874980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.874880    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfd33491-23e6-4e59-922b-dd99c527b119" (UID: "cfd33491-23e6-4e59-922b-dd99c527b119"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:21.875208 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.875067    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-home\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:21.875208 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.875084    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-model-cache\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:21.875208 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.875094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfd33491-23e6-4e59-922b-dd99c527b119" (UID: "cfd33491-23e6-4e59-922b-dd99c527b119"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:21.876871 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.876848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-dshm" (OuterVolumeSpecName: "dshm") pod "cfd33491-23e6-4e59-922b-dd99c527b119" (UID: "cfd33491-23e6-4e59-922b-dd99c527b119"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:21.876991 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.876964    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd33491-23e6-4e59-922b-dd99c527b119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfd33491-23e6-4e59-922b-dd99c527b119" (UID: "cfd33491-23e6-4e59-922b-dd99c527b119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:46:21.877055 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.877001    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd33491-23e6-4e59-922b-dd99c527b119-kube-api-access-n27qj" (OuterVolumeSpecName: "kube-api-access-n27qj") pod "cfd33491-23e6-4e59-922b-dd99c527b119" (UID: "cfd33491-23e6-4e59-922b-dd99c527b119"). InnerVolumeSpecName "kube-api-access-n27qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:46:21.976332 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.976297    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27qj\" (UniqueName: \"kubernetes.io/projected/cfd33491-23e6-4e59-922b-dd99c527b119-kube-api-access-n27qj\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:21.976332 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.976325    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-tmp-dir\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:21.976332 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.976336    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd33491-23e6-4e59-922b-dd99c527b119-dshm\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:21.976590 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:21.976343    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd33491-23e6-4e59-922b-dd99c527b119-tls-certs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.616201 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:22.616169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455" event={"ID":"cfd33491-23e6-4e59-922b-dd99c527b119","Type":"ContainerDied","Data":"04a8ef7681c57c60caa5f1cf05ad5a1f1143239d40391b9291de26b368e594e6"}
Jul 16 22:46:22.616671 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:22.616225    2562 scope.go:117] "RemoveContainer" containerID="6ccf4e7c91a2c2adcc50c00c3d9e0466681dbd63a8b8b95d399606d9e1a4d8ba"
Jul 16 22:46:22.616671 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:22.616240    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"
Jul 16 22:46:22.638837 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:22.638812    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"]
Jul 16 22:46:22.642741 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:22.642720    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/sub-test-model-fbed9b-kserve-68fd648b99-mt455"]
Jul 16 22:46:23.750398 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:23.750359    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd33491-23e6-4e59-922b-dd99c527b119" path="/var/lib/kubelet/pods/cfd33491-23e6-4e59-922b-dd99c527b119/volumes"
Jul 16 22:46:28.759782 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:28.759736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:28.760277 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:28.759802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:28.772153 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:28.772123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:29.662076 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:29.662041    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:46:37.846779 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.846746    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"]
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847078    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847089    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847100    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847106    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847113    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd33491-23e6-4e59-922b-dd99c527b119" containerName="main"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847118    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd33491-23e6-4e59-922b-dd99c527b119" containerName="main"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847179    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28768b3b-8d00-4f9a-9d12-6e5d6fd455f1" containerName="revoke-keys"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847190    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd33491-23e6-4e59-922b-dd99c527b119" containerName="main"
Jul 16 22:46:37.847272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.847195    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25af7d9b-187b-41aa-94da-f58ae1ffdbfa" containerName="revoke-keys"
Jul 16 22:46:37.852147 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.852124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:37.857637 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.857612    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"default-dockercfg-25fwx\""
Jul 16 22:46:37.857805 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.857776    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"kube-root-ca.crt\""
Jul 16 22:46:37.857913 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.857664    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"rate-test-model-106c85-kserve-self-signed-certs\""
Jul 16 22:46:37.857995 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.857612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"openshift-service-ca.crt\""
Jul 16 22:46:37.864310 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.864288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"]
Jul 16 22:46:37.918835 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:37.918800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:38.025265 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.025225    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-model-cache\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.025265 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.025272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-tmp-dir\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.025488 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.025299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-home\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.025488 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.025360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzbs\" (UniqueName: \"kubernetes.io/projected/01b99f23-2177-4658-b65d-8651a2cb3090-kube-api-access-8bzbs\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.025488 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.025393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b99f23-2177-4658-b65d-8651a2cb3090-tls-certs\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.025488 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.025420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-dshm\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126314 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-home\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126314 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzbs\" (UniqueName: \"kubernetes.io/projected/01b99f23-2177-4658-b65d-8651a2cb3090-kube-api-access-8bzbs\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126314 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b99f23-2177-4658-b65d-8651a2cb3090-tls-certs\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126672 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-dshm\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126672 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-model-cache\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126672 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-tmp-dir\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126841 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-home\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126841 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-tmp-dir\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.126921 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.126855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-model-cache\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.128811 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.128788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-dshm\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.129025 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.129009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b99f23-2177-4658-b65d-8651a2cb3090-tls-certs\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.137511 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.137471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzbs\" (UniqueName: \"kubernetes.io/projected/01b99f23-2177-4658-b65d-8651a2cb3090-kube-api-access-8bzbs\") pod \"rate-test-model-106c85-kserve-7bfd464b75-xntpw\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") " pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.161381 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.161355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:38.331092 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.331068    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"]
Jul 16 22:46:38.333755 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:46:38.333725    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01b99f23_2177_4658_b65d_8651a2cb3090.slice/crio-82592296d928b399d55c7e29ce3bc2625a81be91be303ed98cab3bb7a22975f7 WatchSource:0}: Error finding container 82592296d928b399d55c7e29ce3bc2625a81be91be303ed98cab3bb7a22975f7: Status 404 returned error can't find the container with id 82592296d928b399d55c7e29ce3bc2625a81be91be303ed98cab3bb7a22975f7
Jul 16 22:46:38.695784 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.695748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" event={"ID":"01b99f23-2177-4658-b65d-8651a2cb3090","Type":"ContainerStarted","Data":"8834a239bc960689989a2e600dc7abb000d178f5c96ff5cbf74196ec636f6a53"}
Jul 16 22:46:38.695784 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.695787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" event={"ID":"01b99f23-2177-4658-b65d-8651a2cb3090","Type":"ContainerStarted","Data":"82592296d928b399d55c7e29ce3bc2625a81be91be303ed98cab3bb7a22975f7"}
Jul 16 22:46:38.739273 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:38.739223    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" podStartSLOduration=1.739209331 podStartE2EDuration="1.739209331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:38.738820499 +0000 UTC m=+3581.520718350" watchObservedRunningTime="2026-07-16 22:46:38.739209331 +0000 UTC m=+3581.521107179"
Jul 16 22:46:48.162452 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:48.162357    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:48.162452 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:48.162414    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:48.174906 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:48.174882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:48.759749 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:48.759718    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:46:55.761801 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:55.761770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:55.822034 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:46:55.821984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:04.252210 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:04.252172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:04.350172 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:04.350133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:04.598137 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:04.598048    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:12.341215 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:12.341180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:12.444163 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:12.444123    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:22.794308 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:22.794269    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"]
Jul 16 22:47:22.813870 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:22.813827    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:23.715580 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:23.715521    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"]
Jul 16 22:47:23.741283 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:23.741251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:37.814949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:37.814887    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5" podUID="b5b312cc-a830-4334-b2ff-e3c533f0840e" containerName="main" containerID="cri-o://e3778d1eedd0a08ef9d69727cc9fbc9e1a377f926dc04aa861f236c1324b816c" gracePeriod=45
Jul 16 22:47:37.979217 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:37.979176    2562 generic.go:356] "Generic (PLEG): container finished" podID="b5b312cc-a830-4334-b2ff-e3c533f0840e" containerID="e3778d1eedd0a08ef9d69727cc9fbc9e1a377f926dc04aa861f236c1324b816c" exitCode=0
Jul 16 22:47:37.979369 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:37.979245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5" event={"ID":"b5b312cc-a830-4334-b2ff-e3c533f0840e","Type":"ContainerDied","Data":"e3778d1eedd0a08ef9d69727cc9fbc9e1a377f926dc04aa861f236c1324b816c"}
Jul 16 22:47:38.070284 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.070225    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:47:38.180305 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-tmp-dir\") pod \"b5b312cc-a830-4334-b2ff-e3c533f0840e\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") "
Jul 16 22:47:38.180507 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b312cc-a830-4334-b2ff-e3c533f0840e-tls-certs\") pod \"b5b312cc-a830-4334-b2ff-e3c533f0840e\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") "
Jul 16 22:47:38.180507 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180359    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr7zg\" (UniqueName: \"kubernetes.io/projected/b5b312cc-a830-4334-b2ff-e3c533f0840e-kube-api-access-rr7zg\") pod \"b5b312cc-a830-4334-b2ff-e3c533f0840e\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") "
Jul 16 22:47:38.180507 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180381    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-home\") pod \"b5b312cc-a830-4334-b2ff-e3c533f0840e\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") "
Jul 16 22:47:38.180507 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180429    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-dshm\") pod \"b5b312cc-a830-4334-b2ff-e3c533f0840e\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") "
Jul 16 22:47:38.180507 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180468    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-model-cache\") pod \"b5b312cc-a830-4334-b2ff-e3c533f0840e\" (UID: \"b5b312cc-a830-4334-b2ff-e3c533f0840e\") "
Jul 16 22:47:38.180807 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180660    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5b312cc-a830-4334-b2ff-e3c533f0840e" (UID: "b5b312cc-a830-4334-b2ff-e3c533f0840e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:38.180807 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-home" (OuterVolumeSpecName: "home") pod "b5b312cc-a830-4334-b2ff-e3c533f0840e" (UID: "b5b312cc-a830-4334-b2ff-e3c533f0840e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:38.180893 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.180843    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5b312cc-a830-4334-b2ff-e3c533f0840e" (UID: "b5b312cc-a830-4334-b2ff-e3c533f0840e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:38.182796 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.182768    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b312cc-a830-4334-b2ff-e3c533f0840e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5b312cc-a830-4334-b2ff-e3c533f0840e" (UID: "b5b312cc-a830-4334-b2ff-e3c533f0840e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:47:38.182908 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.182795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-dshm" (OuterVolumeSpecName: "dshm") pod "b5b312cc-a830-4334-b2ff-e3c533f0840e" (UID: "b5b312cc-a830-4334-b2ff-e3c533f0840e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:38.182908 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.182809    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b312cc-a830-4334-b2ff-e3c533f0840e-kube-api-access-rr7zg" (OuterVolumeSpecName: "kube-api-access-rr7zg") pod "b5b312cc-a830-4334-b2ff-e3c533f0840e" (UID: "b5b312cc-a830-4334-b2ff-e3c533f0840e"). InnerVolumeSpecName "kube-api-access-rr7zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:47:38.281854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.281815    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-model-cache\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:38.281854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.281847    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-tmp-dir\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:38.281854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.281855    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b312cc-a830-4334-b2ff-e3c533f0840e-tls-certs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:38.281854 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.281865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7zg\" (UniqueName: \"kubernetes.io/projected/b5b312cc-a830-4334-b2ff-e3c533f0840e-kube-api-access-rr7zg\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:38.282146 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.281875    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-home\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:38.282146 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.281883    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b312cc-a830-4334-b2ff-e3c533f0840e-dshm\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:38.735688 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.735625    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" podUID="01b99f23-2177-4658-b65d-8651a2cb3090" containerName="main" containerID="cri-o://8834a239bc960689989a2e600dc7abb000d178f5c96ff5cbf74196ec636f6a53" gracePeriod=45
Jul 16 22:47:38.748913 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.748874    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" podUID="01b99f23-2177-4658-b65d-8651a2cb3090" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 16 22:47:38.986239 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.986172    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"
Jul 16 22:47:38.986239 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.986203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5" event={"ID":"b5b312cc-a830-4334-b2ff-e3c533f0840e","Type":"ContainerDied","Data":"e31438b40809ec4b59300a627e938f8cd0f0e11731173b7064f4d807ea1ace2e"}
Jul 16 22:47:38.986694 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.986263    2562 scope.go:117] "RemoveContainer" containerID="e3778d1eedd0a08ef9d69727cc9fbc9e1a377f926dc04aa861f236c1324b816c"
Jul 16 22:47:38.987923 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.987873    2562 generic.go:356] "Generic (PLEG): container finished" podID="01b99f23-2177-4658-b65d-8651a2cb3090" containerID="8834a239bc960689989a2e600dc7abb000d178f5c96ff5cbf74196ec636f6a53" exitCode=0
Jul 16 22:47:38.988031 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.987965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" event={"ID":"01b99f23-2177-4658-b65d-8651a2cb3090","Type":"ContainerDied","Data":"8834a239bc960689989a2e600dc7abb000d178f5c96ff5cbf74196ec636f6a53"}
Jul 16 22:47:38.988031 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.988007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw" event={"ID":"01b99f23-2177-4658-b65d-8651a2cb3090","Type":"ContainerDied","Data":"82592296d928b399d55c7e29ce3bc2625a81be91be303ed98cab3bb7a22975f7"}
Jul 16 22:47:38.988031 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.988021    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82592296d928b399d55c7e29ce3bc2625a81be91be303ed98cab3bb7a22975f7"
Jul 16 22:47:38.991949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:38.991930    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:47:39.025087 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.025055    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"]
Jul 16 22:47:39.028980 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.028957    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/rate-test-model-fbed9b-kserve-79fb7cf4c5-6jhp5"]
Jul 16 22:47:39.088437 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088406    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bzbs\" (UniqueName: \"kubernetes.io/projected/01b99f23-2177-4658-b65d-8651a2cb3090-kube-api-access-8bzbs\") pod \"01b99f23-2177-4658-b65d-8651a2cb3090\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") "
Jul 16 22:47:39.088632 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088454    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-home\") pod \"01b99f23-2177-4658-b65d-8651a2cb3090\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") "
Jul 16 22:47:39.088632 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-model-cache\") pod \"01b99f23-2177-4658-b65d-8651a2cb3090\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") "
Jul 16 22:47:39.088632 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088537    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-tmp-dir\") pod \"01b99f23-2177-4658-b65d-8651a2cb3090\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") "
Jul 16 22:47:39.088632 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088577    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-dshm\") pod \"01b99f23-2177-4658-b65d-8651a2cb3090\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") "
Jul 16 22:47:39.088832 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088675    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b99f23-2177-4658-b65d-8651a2cb3090-tls-certs\") pod \"01b99f23-2177-4658-b65d-8651a2cb3090\" (UID: \"01b99f23-2177-4658-b65d-8651a2cb3090\") "
Jul 16 22:47:39.088832 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-model-cache" (OuterVolumeSpecName: "model-cache") pod "01b99f23-2177-4658-b65d-8651a2cb3090" (UID: "01b99f23-2177-4658-b65d-8651a2cb3090"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:39.088832 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-home" (OuterVolumeSpecName: "home") pod "01b99f23-2177-4658-b65d-8651a2cb3090" (UID: "01b99f23-2177-4658-b65d-8651a2cb3090"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:39.088982 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088841    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01b99f23-2177-4658-b65d-8651a2cb3090" (UID: "01b99f23-2177-4658-b65d-8651a2cb3090"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:39.089046 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.088989    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-home\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:39.089046 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.089007    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-model-cache\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:39.089046 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.089017    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-tmp-dir\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:39.090808 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.090782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-dshm" (OuterVolumeSpecName: "dshm") pod "01b99f23-2177-4658-b65d-8651a2cb3090" (UID: "01b99f23-2177-4658-b65d-8651a2cb3090"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:39.090916 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.090861    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b99f23-2177-4658-b65d-8651a2cb3090-kube-api-access-8bzbs" (OuterVolumeSpecName: "kube-api-access-8bzbs") pod "01b99f23-2177-4658-b65d-8651a2cb3090" (UID: "01b99f23-2177-4658-b65d-8651a2cb3090"). InnerVolumeSpecName "kube-api-access-8bzbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:47:39.090916 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.090865    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b99f23-2177-4658-b65d-8651a2cb3090-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01b99f23-2177-4658-b65d-8651a2cb3090" (UID: "01b99f23-2177-4658-b65d-8651a2cb3090"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:47:39.189747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.189715    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b99f23-2177-4658-b65d-8651a2cb3090-tls-certs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:39.189747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.189742    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzbs\" (UniqueName: \"kubernetes.io/projected/01b99f23-2177-4658-b65d-8651a2cb3090-kube-api-access-8bzbs\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:39.189747 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.189751    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b99f23-2177-4658-b65d-8651a2cb3090-dshm\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:39.750757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.750723    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b312cc-a830-4334-b2ff-e3c533f0840e" path="/var/lib/kubelet/pods/b5b312cc-a830-4334-b2ff-e3c533f0840e/volumes"
Jul 16 22:47:39.993800 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:39.993775    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"
Jul 16 22:47:40.015656 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:40.015574    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"]
Jul 16 22:47:40.018853 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:40.018821    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/rate-test-model-106c85-kserve-7bfd464b75-xntpw"]
Jul 16 22:47:41.751359 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:41.751328    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b99f23-2177-4658-b65d-8651a2cb3090" path="/var/lib/kubelet/pods/01b99f23-2177-4658-b65d-8651a2cb3090/volumes"
Jul 16 22:47:43.277233 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277193    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"]
Jul 16 22:47:43.277748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277529    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b99f23-2177-4658-b65d-8651a2cb3090" containerName="main"
Jul 16 22:47:43.277748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277540    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b99f23-2177-4658-b65d-8651a2cb3090" containerName="main"
Jul 16 22:47:43.277748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277566    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b312cc-a830-4334-b2ff-e3c533f0840e" containerName="main"
Jul 16 22:47:43.277748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277572    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b312cc-a830-4334-b2ff-e3c533f0840e" containerName="main"
Jul 16 22:47:43.277748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277637    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b312cc-a830-4334-b2ff-e3c533f0840e" containerName="main"
Jul 16 22:47:43.277748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.277647    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b99f23-2177-4658-b65d-8651a2cb3090" containerName="main"
Jul 16 22:47:43.280700 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.280677    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:43.283145 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.283123    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:47:43.288860 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.288832    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"]
Jul 16 22:47:43.424206 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.424177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a139c-4afb-44d4-8576-5974d3331fa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br\" (UID: \"4a9a139c-4afb-44d4-8576-5974d3331fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:43.525768 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.525732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a139c-4afb-44d4-8576-5974d3331fa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br\" (UID: \"4a9a139c-4afb-44d4-8576-5974d3331fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:43.526335 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.526317    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a139c-4afb-44d4-8576-5974d3331fa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br\" (UID: \"4a9a139c-4afb-44d4-8576-5974d3331fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:43.591904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.591823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:43.733092 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.733068    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"]
Jul 16 22:47:43.735620 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:47:43.735588    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9a139c_4afb_44d4_8576_5974d3331fa8.slice/crio-ecd244306728160bb6e9d69e1dbdd4cad7f1eb06a2876ccb139c08ec32dd4ae5 WatchSource:0}: Error finding container ecd244306728160bb6e9d69e1dbdd4cad7f1eb06a2876ccb139c08ec32dd4ae5: Status 404 returned error can't find the container with id ecd244306728160bb6e9d69e1dbdd4cad7f1eb06a2876ccb139c08ec32dd4ae5
Jul 16 22:47:43.889334 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.889250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:43.989888 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:43.989851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:44.014382 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:44.014350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br" event={"ID":"4a9a139c-4afb-44d4-8576-5974d3331fa8","Type":"ContainerStarted","Data":"ecd244306728160bb6e9d69e1dbdd4cad7f1eb06a2876ccb139c08ec32dd4ae5"}
Jul 16 22:47:45.020272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:45.020240    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a9a139c-4afb-44d4-8576-5974d3331fa8" containerID="a770ef7f00c4d5a9c5d3e99534775c08ec2f11b1ac68ab0d6a126a8d80b900fa" exitCode=0
Jul 16 22:47:45.020272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:45.020276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br" event={"ID":"4a9a139c-4afb-44d4-8576-5974d3331fa8","Type":"ContainerDied","Data":"a770ef7f00c4d5a9c5d3e99534775c08ec2f11b1ac68ab0d6a126a8d80b900fa"}
Jul 16 22:47:46.157741 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:46.157717    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:46.250272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:46.250231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a139c-4afb-44d4-8576-5974d3331fa8-maas-api-service-ca\") pod \"4a9a139c-4afb-44d4-8576-5974d3331fa8\" (UID: \"4a9a139c-4afb-44d4-8576-5974d3331fa8\") "
Jul 16 22:47:46.250524 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:46.250504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9a139c-4afb-44d4-8576-5974d3331fa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a9a139c-4afb-44d4-8576-5974d3331fa8" (UID: "4a9a139c-4afb-44d4-8576-5974d3331fa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:46.351630 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:46.351528    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a139c-4afb-44d4-8576-5974d3331fa8-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:47.031034 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:47.031004    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br"
Jul 16 22:47:47.031240 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:47.031038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br" event={"ID":"4a9a139c-4afb-44d4-8576-5974d3331fa8","Type":"ContainerDied","Data":"ecd244306728160bb6e9d69e1dbdd4cad7f1eb06a2876ccb139c08ec32dd4ae5"}
Jul 16 22:47:47.031240 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:47.031090    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecd244306728160bb6e9d69e1dbdd4cad7f1eb06a2876ccb139c08ec32dd4ae5"
Jul 16 22:47:53.799816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:53.798866    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"]
Jul 16 22:47:53.799816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:53.799180    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" podUID="46da0e33-7e1d-45f2-ba29-4f9e225abccb" containerName="maas-api" containerID="cri-o://42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e" gracePeriod=30
Jul 16 22:47:53.828457 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:53.828415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:54.050889 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.050831    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:47:54.065874 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.065845    2562 generic.go:356] "Generic (PLEG): container finished" podID="46da0e33-7e1d-45f2-ba29-4f9e225abccb" containerID="42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e" exitCode=0
Jul 16 22:47:54.066038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.065888    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"
Jul 16 22:47:54.066038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.065921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" event={"ID":"46da0e33-7e1d-45f2-ba29-4f9e225abccb","Type":"ContainerDied","Data":"42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e"}
Jul 16 22:47:54.066038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.065968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr" event={"ID":"46da0e33-7e1d-45f2-ba29-4f9e225abccb","Type":"ContainerDied","Data":"bcec604834db0f610800e0c66cc07dfb12c922bd606ab7b34618597e0b61a2b9"}
Jul 16 22:47:54.066038 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.065998    2562 scope.go:117] "RemoveContainer" containerID="42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e"
Jul 16 22:47:54.076827 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.076806    2562 scope.go:117] "RemoveContainer" containerID="42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e"
Jul 16 22:47:54.077172 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:47:54.077145    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e\": container with ID starting with 42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e not found: ID does not exist" containerID="42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e"
Jul 16 22:47:54.077282 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.077181    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e"} err="failed to get container status \"42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e\": rpc error: code = NotFound desc = could not find container \"42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e\": container with ID starting with 42dca629b96c0c0d3ff6441021c27697cfeda723231fbd46174a153345a3232e not found: ID does not exist"
Jul 16 22:47:54.220058 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.220026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls\") pod \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") "
Jul 16 22:47:54.220272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.220076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46da0e33-7e1d-45f2-ba29-4f9e225abccb-openshift-service-ca\") pod \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") "
Jul 16 22:47:54.220272 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.220117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rngh8\" (UniqueName: \"kubernetes.io/projected/46da0e33-7e1d-45f2-ba29-4f9e225abccb-kube-api-access-rngh8\") pod \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\" (UID: \"46da0e33-7e1d-45f2-ba29-4f9e225abccb\") "
Jul 16 22:47:54.220538 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.220510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46da0e33-7e1d-45f2-ba29-4f9e225abccb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46da0e33-7e1d-45f2-ba29-4f9e225abccb" (UID: "46da0e33-7e1d-45f2-ba29-4f9e225abccb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:54.222271 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.222247    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46da0e33-7e1d-45f2-ba29-4f9e225abccb" (UID: "46da0e33-7e1d-45f2-ba29-4f9e225abccb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:47:54.222366 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.222342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46da0e33-7e1d-45f2-ba29-4f9e225abccb-kube-api-access-rngh8" (OuterVolumeSpecName: "kube-api-access-rngh8") pod "46da0e33-7e1d-45f2-ba29-4f9e225abccb" (UID: "46da0e33-7e1d-45f2-ba29-4f9e225abccb"). InnerVolumeSpecName "kube-api-access-rngh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:47:54.321056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.320972    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46da0e33-7e1d-45f2-ba29-4f9e225abccb-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:54.321056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.320999    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46da0e33-7e1d-45f2-ba29-4f9e225abccb-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:54.321056 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.321009    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngh8\" (UniqueName: \"kubernetes.io/projected/46da0e33-7e1d-45f2-ba29-4f9e225abccb-kube-api-access-rngh8\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:47:54.387665 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.387635    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"]
Jul 16 22:47:54.393131 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:54.393108    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fbed9b-f4c84b4bf-z7kzr"]
Jul 16 22:47:55.751144 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:55.751110    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46da0e33-7e1d-45f2-ba29-4f9e225abccb" path="/var/lib/kubelet/pods/46da0e33-7e1d-45f2-ba29-4f9e225abccb/volumes"
Jul 16 22:47:58.389305 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:47:58.389265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:04.828468 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:04.828432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:04.932598 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:04.932532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:09.151901 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.151860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:09.379362 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.379329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:09.518534 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518413    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"]
Jul 16 22:48:09.518832 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518818    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9a139c-4afb-44d4-8576-5974d3331fa8" containerName="revoke-keys"
Jul 16 22:48:09.518883 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518837    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9a139c-4afb-44d4-8576-5974d3331fa8" containerName="revoke-keys"
Jul 16 22:48:09.518883 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518861    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46da0e33-7e1d-45f2-ba29-4f9e225abccb" containerName="maas-api"
Jul 16 22:48:09.518883 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518867    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="46da0e33-7e1d-45f2-ba29-4f9e225abccb" containerName="maas-api"
Jul 16 22:48:09.518983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518924    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9a139c-4afb-44d4-8576-5974d3331fa8" containerName="revoke-keys"
Jul 16 22:48:09.518983 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.518933    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="46da0e33-7e1d-45f2-ba29-4f9e225abccb" containerName="maas-api"
Jul 16 22:48:09.524017 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.524002    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:09.526714 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.526694    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:48:09.531209 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.531184    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"]
Jul 16 22:48:09.554957 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.554923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1393af64-43f0-4ae0-b227-b0f95313bff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk\" (UID: \"1393af64-43f0-4ae0-b227-b0f95313bff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:09.655887 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.655856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1393af64-43f0-4ae0-b227-b0f95313bff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk\" (UID: \"1393af64-43f0-4ae0-b227-b0f95313bff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:09.656482 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.656461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1393af64-43f0-4ae0-b227-b0f95313bff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk\" (UID: \"1393af64-43f0-4ae0-b227-b0f95313bff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:09.836584 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.836484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:09.974693 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:09.974663    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"]
Jul 16 22:48:09.977574 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:48:09.977531    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1393af64_43f0_4ae0_b227_b0f95313bff2.slice/crio-e173b41d34adf9e661bafd73d9210d6235eff25aeae844e86b7c858a8153102b WatchSource:0}: Error finding container e173b41d34adf9e661bafd73d9210d6235eff25aeae844e86b7c858a8153102b: Status 404 returned error can't find the container with id e173b41d34adf9e661bafd73d9210d6235eff25aeae844e86b7c858a8153102b
Jul 16 22:48:10.146260 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:10.146179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk" event={"ID":"1393af64-43f0-4ae0-b227-b0f95313bff2","Type":"ContainerStarted","Data":"e173b41d34adf9e661bafd73d9210d6235eff25aeae844e86b7c858a8153102b"}
Jul 16 22:48:10.254902 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:10.254876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:10.351720 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:10.351688    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:11.152644 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:11.152610    2562 generic.go:356] "Generic (PLEG): container finished" podID="1393af64-43f0-4ae0-b227-b0f95313bff2" containerID="1aa3a7594c81720c4db7ff47ca4d0863048158640be35317814d3d33a0e5c898" exitCode=0
Jul 16 22:48:11.152846 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:11.152651    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk" event={"ID":"1393af64-43f0-4ae0-b227-b0f95313bff2","Type":"ContainerDied","Data":"1aa3a7594c81720c4db7ff47ca4d0863048158640be35317814d3d33a0e5c898"}
Jul 16 22:48:12.292708 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:12.292683    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:12.382253 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:12.382213    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1393af64-43f0-4ae0-b227-b0f95313bff2-maas-api-service-ca\") pod \"1393af64-43f0-4ae0-b227-b0f95313bff2\" (UID: \"1393af64-43f0-4ae0-b227-b0f95313bff2\") "
Jul 16 22:48:12.382612 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:12.382590    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1393af64-43f0-4ae0-b227-b0f95313bff2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1393af64-43f0-4ae0-b227-b0f95313bff2" (UID: "1393af64-43f0-4ae0-b227-b0f95313bff2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:12.483714 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:12.483611    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1393af64-43f0-4ae0-b227-b0f95313bff2-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:48:13.163495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:13.163465    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk"
Jul 16 22:48:13.163686 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:13.163495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk" event={"ID":"1393af64-43f0-4ae0-b227-b0f95313bff2","Type":"ContainerDied","Data":"e173b41d34adf9e661bafd73d9210d6235eff25aeae844e86b7c858a8153102b"}
Jul 16 22:48:13.163686 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:13.163582    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e173b41d34adf9e661bafd73d9210d6235eff25aeae844e86b7c858a8153102b"
Jul 16 22:48:20.060067 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.059989    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"]
Jul 16 22:48:20.060454 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.060220    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" podUID="aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" containerName="maas-api" containerID="cri-o://f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25" gracePeriod=30
Jul 16 22:48:20.091689 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.091651    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:20.502148 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.502122    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:48:20.554327 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.554299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrdlv\" (UniqueName: \"kubernetes.io/projected/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-kube-api-access-qrdlv\") pod \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") "
Jul 16 22:48:20.554495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.554348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-openshift-service-ca\") pod \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") "
Jul 16 22:48:20.554495 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.554375    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls\") pod \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\" (UID: \"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077\") "
Jul 16 22:48:20.554748 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.554723    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" (UID: "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:20.556527 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.556499    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-kube-api-access-qrdlv" (OuterVolumeSpecName: "kube-api-access-qrdlv") pod "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" (UID: "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077"). InnerVolumeSpecName "kube-api-access-qrdlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:48:20.556527 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.556521    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" (UID: "aaeb5cbd-d791-4f13-8e5c-48c77dc4d077"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:48:20.655816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.655778    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-maas-api-tls\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:48:20.655816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.655810    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdlv\" (UniqueName: \"kubernetes.io/projected/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-kube-api-access-qrdlv\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:48:20.655816 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:20.655821    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077-openshift-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:48:21.202329 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.202287    2562 generic.go:356] "Generic (PLEG): container finished" podID="aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" containerID="f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25" exitCode=0
Jul 16 22:48:21.202825 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.202335    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"
Jul 16 22:48:21.202825 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.202379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" event={"ID":"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077","Type":"ContainerDied","Data":"f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25"}
Jul 16 22:48:21.202825 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.202437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps" event={"ID":"aaeb5cbd-d791-4f13-8e5c-48c77dc4d077","Type":"ContainerDied","Data":"e2dffc141ac59070ab20e6f9854a83d489efdaf3816cf7d9891f1856c5bf584d"}
Jul 16 22:48:21.202825 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.202464    2562 scope.go:117] "RemoveContainer" containerID="f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25"
Jul 16 22:48:21.212383 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.212366    2562 scope.go:117] "RemoveContainer" containerID="f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25"
Jul 16 22:48:21.212699 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:48:21.212681    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25\": container with ID starting with f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25 not found: ID does not exist" containerID="f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25"
Jul 16 22:48:21.212759 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.212709    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25"} err="failed to get container status \"f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25\": rpc error: code = NotFound desc = could not find container \"f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25\": container with ID starting with f797f370d7937bca8c5ed75336be11ead170fde12325860962cc9e842f089e25 not found: ID does not exist"
Jul 16 22:48:21.224964 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.224940    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"]
Jul 16 22:48:21.232377 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.232354    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-106c85-855ddbff65-b7bps"]
Jul 16 22:48:21.750977 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:21.750942    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" path="/var/lib/kubelet/pods/aaeb5cbd-d791-4f13-8e5c-48c77dc4d077/volumes"
Jul 16 22:48:24.593084 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:24.593045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:26.415523 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415489    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"]
Jul 16 22:48:26.416049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415851    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1393af64-43f0-4ae0-b227-b0f95313bff2" containerName="revoke-keys"
Jul 16 22:48:26.416049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415868    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1393af64-43f0-4ae0-b227-b0f95313bff2" containerName="revoke-keys"
Jul 16 22:48:26.416049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415884    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" containerName="maas-api"
Jul 16 22:48:26.416049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415890    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" containerName="maas-api"
Jul 16 22:48:26.416049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415941    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1393af64-43f0-4ae0-b227-b0f95313bff2" containerName="revoke-keys"
Jul 16 22:48:26.416049 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.415952    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaeb5cbd-d791-4f13-8e5c-48c77dc4d077" containerName="maas-api"
Jul 16 22:48:26.420206 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.420189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:26.424678 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.424659    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:48:26.424785 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.424663    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:48:26.435511 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.435486    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"]
Jul 16 22:48:26.508949 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.508912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed66022-66ee-49cb-85e1-6363db5e19a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr\" (UID: \"7ed66022-66ee-49cb-85e1-6363db5e19a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:26.610290 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.610257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed66022-66ee-49cb-85e1-6363db5e19a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr\" (UID: \"7ed66022-66ee-49cb-85e1-6363db5e19a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:26.610906 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.610885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed66022-66ee-49cb-85e1-6363db5e19a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr\" (UID: \"7ed66022-66ee-49cb-85e1-6363db5e19a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:26.730708 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.730611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:26.865267 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:26.865239    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"]
Jul 16 22:48:26.867323 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:48:26.867283    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed66022_66ee_49cb_85e1_6363db5e19a7.slice/crio-d48094dc8143a8fafb2f5dfd4a9097b67efb5b8e463c4b136efe3bd7a5bfcb2d WatchSource:0}: Error finding container d48094dc8143a8fafb2f5dfd4a9097b67efb5b8e463c4b136efe3bd7a5bfcb2d: Status 404 returned error can't find the container with id d48094dc8143a8fafb2f5dfd4a9097b67efb5b8e463c4b136efe3bd7a5bfcb2d
Jul 16 22:48:27.242861 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:27.242824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerStarted","Data":"0070fe18ec9d6a7e411a196f0d6d0439924526c843d9d1713d2fa523450b8f4c"}
Jul 16 22:48:27.243029 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:27.242882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerStarted","Data":"d48094dc8143a8fafb2f5dfd4a9097b67efb5b8e463c4b136efe3bd7a5bfcb2d"}
Jul 16 22:48:27.260334 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:27.260278    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" podStartSLOduration=1.260263726 podStartE2EDuration="1.260263726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:27.259272935 +0000 UTC m=+3690.041170786" watchObservedRunningTime="2026-07-16 22:48:27.260263726 +0000 UTC m=+3690.042161573"
Jul 16 22:48:28.249372 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:28.249333    2562 generic.go:356] "Generic (PLEG): container finished" podID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerID="0070fe18ec9d6a7e411a196f0d6d0439924526c843d9d1713d2fa523450b8f4c" exitCode=6
Jul 16 22:48:28.249892 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:28.249414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerDied","Data":"0070fe18ec9d6a7e411a196f0d6d0439924526c843d9d1713d2fa523450b8f4c"}
Jul 16 22:48:28.249892 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:28.249696    2562 scope.go:117] "RemoveContainer" containerID="0070fe18ec9d6a7e411a196f0d6d0439924526c843d9d1713d2fa523450b8f4c"
Jul 16 22:48:29.255621 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:29.255581    2562 generic.go:356] "Generic (PLEG): container finished" podID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerID="ada0a27c877896c6852bbb3d7fabd6995ca22d5bc0d105a424d6a4f8cf5959b7" exitCode=6
Jul 16 22:48:29.256114 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:29.255673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerDied","Data":"ada0a27c877896c6852bbb3d7fabd6995ca22d5bc0d105a424d6a4f8cf5959b7"}
Jul 16 22:48:29.256114 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:29.255748    2562 scope.go:117] "RemoveContainer" containerID="0070fe18ec9d6a7e411a196f0d6d0439924526c843d9d1713d2fa523450b8f4c"
Jul 16 22:48:29.256114 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:29.255974    2562 scope.go:117] "RemoveContainer" containerID="ada0a27c877896c6852bbb3d7fabd6995ca22d5bc0d105a424d6a4f8cf5959b7"
Jul 16 22:48:29.256247 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:48:29.256200    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr_odh-ai-gateway-infra(7ed66022-66ee-49cb-85e1-6363db5e19a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7"
Jul 16 22:48:30.262136 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:30.262106    2562 scope.go:117] "RemoveContainer" containerID="ada0a27c877896c6852bbb3d7fabd6995ca22d5bc0d105a424d6a4f8cf5959b7"
Jul 16 22:48:30.262516 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:48:30.262295    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr_odh-ai-gateway-infra(7ed66022-66ee-49cb-85e1-6363db5e19a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7"
Jul 16 22:48:31.675450 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:31.675413    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:31.774619 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:31.774588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:35.298191 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:35.298159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:35.527023 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:35.526991    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:41.592473 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:41.592441    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-95466d4fd-mdsdq_c5f395d0-76cf-47bf-a1c4-1e15a9ed061c/manager/0.log"
Jul 16 22:48:42.213373 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:42.213344    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-vt7sp_9debcaac-f977-4b23-840d-81dd2ab761d8/copy-manifests/0.log"
Jul 16 22:48:42.244746 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:42.244712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-vt7sp_9debcaac-f977-4b23-840d-81dd2ab761d8/manager/0.log"
Jul 16 22:48:42.839563 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:42.839514    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737350-4t7h8_6ab60dd2-1cc9-4d1a-9f9b-4ed70cb99fe7/cleanup/0.log"
Jul 16 22:48:42.948317 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:42.948282    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737365-7dhpt_76f082cb-f2c5-4f15-ad42-2e913bc61eae/cleanup/0.log"
Jul 16 22:48:43.176776 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:43.176747    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-a0061f-67lgv_269e3f20-1a37-450b-ae4b-6ccdfb83dcdb/revoke-keys/0.log"
Jul 16 22:48:43.739792 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:43.739766    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr_7ed66022-66ee-49cb-85e1-6363db5e19a7/revoke-keys/1.log"
Jul 16 22:48:43.745173 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:43.745159    2562 scope.go:117] "RemoveContainer" containerID="ada0a27c877896c6852bbb3d7fabd6995ca22d5bc0d105a424d6a4f8cf5959b7"
Jul 16 22:48:44.062978 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:44.062952    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-fbed9b-sc4br_4a9a139c-4afb-44d4-8576-5974d3331fa8/revoke-keys/0.log"
Jul 16 22:48:44.169314 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:44.169287    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-106c85-s6zfk_1393af64-43f0-4ae0-b227-b0f95313bff2/revoke-keys/0.log"
Jul 16 22:48:44.326571 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:44.326456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerStarted","Data":"b62085a34df24bf6cce2071dee9194a5cafb5e598a71ab4eee2e9dd3478109fe"}
Jul 16 22:48:45.332963 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:45.332929    2562 generic.go:356] "Generic (PLEG): container finished" podID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerID="b62085a34df24bf6cce2071dee9194a5cafb5e598a71ab4eee2e9dd3478109fe" exitCode=6
Jul 16 22:48:45.333406 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:45.332999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerDied","Data":"b62085a34df24bf6cce2071dee9194a5cafb5e598a71ab4eee2e9dd3478109fe"}
Jul 16 22:48:45.333406 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:45.333056    2562 scope.go:117] "RemoveContainer" containerID="ada0a27c877896c6852bbb3d7fabd6995ca22d5bc0d105a424d6a4f8cf5959b7"
Jul 16 22:48:45.333508 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:45.333403    2562 scope.go:117] "RemoveContainer" containerID="b62085a34df24bf6cce2071dee9194a5cafb5e598a71ab4eee2e9dd3478109fe"
Jul 16 22:48:45.333662 ip-10-0-139-156 kubenswrapper[2562]: E0716 22:48:45.333633    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr_odh-ai-gateway-infra(7ed66022-66ee-49cb-85e1-6363db5e19a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7"
Jul 16 22:48:45.358307 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:45.358283    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"]
Jul 16 22:48:45.913294 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:45.913268    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5q7z6_a00aeffd-1a16-4497-a925-b660f84b0976/kuadrant-console-plugin/0.log"
Jul 16 22:48:46.148856 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:46.148827    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lwplb_a0b18074-6556-433d-bd18-4dfb8a52688b/manager/0.log"
Jul 16 22:48:46.477003 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:46.476981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:46.495934 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:46.495906    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed66022-66ee-49cb-85e1-6363db5e19a7-maas-api-service-ca\") pod \"7ed66022-66ee-49cb-85e1-6363db5e19a7\" (UID: \"7ed66022-66ee-49cb-85e1-6363db5e19a7\") "
Jul 16 22:48:46.496468 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:46.496434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed66022-66ee-49cb-85e1-6363db5e19a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ed66022-66ee-49cb-85e1-6363db5e19a7" (UID: "7ed66022-66ee-49cb-85e1-6363db5e19a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:46.597195 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:46.597114    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed66022-66ee-49cb-85e1-6363db5e19a7-maas-api-service-ca\") on node \"ip-10-0-139-156.ec2.internal\" DevicePath \"\""
Jul 16 22:48:46.986891 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:46.986858    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-76tr8_98626467-5488-41b5-a3f5-498667623a37/kube-auth-proxy/0.log"
Jul 16 22:48:47.344148 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.344074    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-r8vkg_a9cf7475-5dcb-40af-8636-1882c45278f3/payload-pre-processing/0.log"
Jul 16 22:48:47.345535 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.345510    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"
Jul 16 22:48:47.345691 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.345539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr" event={"ID":"7ed66022-66ee-49cb-85e1-6363db5e19a7","Type":"ContainerDied","Data":"d48094dc8143a8fafb2f5dfd4a9097b67efb5b8e463c4b136efe3bd7a5bfcb2d"}
Jul 16 22:48:47.345691 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.345638    2562 scope.go:117] "RemoveContainer" containerID="b62085a34df24bf6cce2071dee9194a5cafb5e598a71ab4eee2e9dd3478109fe"
Jul 16 22:48:47.367316 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.367291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"]
Jul 16 22:48:47.373273 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.373250    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7qrhr"]
Jul 16 22:48:47.750087 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:47.750057    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" path="/var/lib/kubelet/pods/7ed66022-66ee-49cb-85e1-6363db5e19a7/volumes"
Jul 16 22:48:48.036734 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:48.036651    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tllmv_f66383d6-fd59-46ce-a7b3-e257c01cca2e/main/0.log"
Jul 16 22:48:48.267634 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:48.267599    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db6zlnt_3483aa9e-7b82-4422-aa94-c702e7eb32d2/main/0.log"
Jul 16 22:48:48.386208 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:48.386140    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6jjvc_e19e3f71-e690-4d4f-aa38-6d17e0f9f592/main/0.log"
Jul 16 22:48:54.303191 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303154    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nx7gf/must-gather-7bndt"]
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303477    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303488    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303509    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303514    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303573    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303582    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303603 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303589    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303833 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303644    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.303833 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.303650    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed66022-66ee-49cb-85e1-6363db5e19a7" containerName="revoke-keys"
Jul 16 22:48:54.307593 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.307575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.322251 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.322233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nx7gf\"/\"default-dockercfg-pvwf8\""
Jul 16 22:48:54.322638 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.322623    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nx7gf\"/\"kube-root-ca.crt\""
Jul 16 22:48:54.328295 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.328270    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nx7gf/must-gather-7bndt"]
Jul 16 22:48:54.334757 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.334741    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nx7gf\"/\"openshift-service-ca.crt\""
Jul 16 22:48:54.351362 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.351331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/002b8616-9ae0-4159-97a3-cfe9d7a579c0-must-gather-output\") pod \"must-gather-7bndt\" (UID: \"002b8616-9ae0-4159-97a3-cfe9d7a579c0\") " pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.351479 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.351380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfbwc\" (UniqueName: \"kubernetes.io/projected/002b8616-9ae0-4159-97a3-cfe9d7a579c0-kube-api-access-tfbwc\") pod \"must-gather-7bndt\" (UID: \"002b8616-9ae0-4159-97a3-cfe9d7a579c0\") " pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.362197 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.362171    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:54.452104 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.452076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/002b8616-9ae0-4159-97a3-cfe9d7a579c0-must-gather-output\") pod \"must-gather-7bndt\" (UID: \"002b8616-9ae0-4159-97a3-cfe9d7a579c0\") " pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.452280 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.452129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfbwc\" (UniqueName: \"kubernetes.io/projected/002b8616-9ae0-4159-97a3-cfe9d7a579c0-kube-api-access-tfbwc\") pod \"must-gather-7bndt\" (UID: \"002b8616-9ae0-4159-97a3-cfe9d7a579c0\") " pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.452403 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.452383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/002b8616-9ae0-4159-97a3-cfe9d7a579c0-must-gather-output\") pod \"must-gather-7bndt\" (UID: \"002b8616-9ae0-4159-97a3-cfe9d7a579c0\") " pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.474587 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.474543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfbwc\" (UniqueName: \"kubernetes.io/projected/002b8616-9ae0-4159-97a3-cfe9d7a579c0-kube-api-access-tfbwc\") pod \"must-gather-7bndt\" (UID: \"002b8616-9ae0-4159-97a3-cfe9d7a579c0\") " pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.500069 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.500040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:54.616088 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.615982    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nx7gf/must-gather-7bndt"
Jul 16 22:48:54.750718 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.750693    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nx7gf/must-gather-7bndt"]
Jul 16 22:48:54.752886 ip-10-0-139-156 kubenswrapper[2562]: W0716 22:48:54.752861    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod002b8616_9ae0_4159_97a3_cfe9d7a579c0.slice/crio-679f0533b865d06ac2f336e0a48b7b467fbb2952697f1ea90cb36da23a83803e WatchSource:0}: Error finding container 679f0533b865d06ac2f336e0a48b7b467fbb2952697f1ea90cb36da23a83803e: Status 404 returned error can't find the container with id 679f0533b865d06ac2f336e0a48b7b467fbb2952697f1ea90cb36da23a83803e
Jul 16 22:48:54.754828 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:54.754810    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:48:55.387452 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:55.387415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/must-gather-7bndt" event={"ID":"002b8616-9ae0-4159-97a3-cfe9d7a579c0","Type":"ContainerStarted","Data":"679f0533b865d06ac2f336e0a48b7b467fbb2952697f1ea90cb36da23a83803e"}
Jul 16 22:48:56.398165 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:56.397331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/must-gather-7bndt" event={"ID":"002b8616-9ae0-4159-97a3-cfe9d7a579c0","Type":"ContainerStarted","Data":"3504f6bc28d253339fd5d57ac853ea2d4bf29e65051f3c2658eb3f40a07cad92"}
Jul 16 22:48:56.398165 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:56.397385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/must-gather-7bndt" event={"ID":"002b8616-9ae0-4159-97a3-cfe9d7a579c0","Type":"ContainerStarted","Data":"aa5e298280f0adb0e4d31e44d85a57576e16b6996e7b01b91b2fcdca3dba24c2"}
Jul 16 22:48:56.418394 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:56.418329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nx7gf/must-gather-7bndt" podStartSLOduration=1.8407789989999999 podStartE2EDuration="2.418305783s" totalImagesPullingTime="577.526784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:56.416918753 +0000 UTC m=+3719.198816601" watchObservedRunningTime="2026-07-16 22:48:56.418305783 +0000 UTC m=+3719.200203632"
Jul 16 22:48:57.040729 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:57.040694    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h25ht_0bec7cf3-d1a3-4497-b829-932fbc703c4a/global-pull-secret-syncer/0.log"
Jul 16 22:48:57.145829 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:57.145794    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4rs4r_52a5b129-e482-4486-a543-a7ae3a011c36/konnectivity-agent/0.log"
Jul 16 22:48:57.304997 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:48:57.304919    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-156.ec2.internal_80f21118e6e677b8e438ca527e429584/haproxy/0.log"
Jul 16 22:49:03.290853 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:03.290815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5q7z6_a00aeffd-1a16-4497-a925-b660f84b0976/kuadrant-console-plugin/0.log"
Jul 16 22:49:03.508573 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:03.507988    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lwplb_a0b18074-6556-433d-bd18-4dfb8a52688b/manager/0.log"
Jul 16 22:49:05.714870 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:05.714804    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pdpx_b8dfe233-77be-496f-9683-9b61adea1438/node-exporter/0.log"
Jul 16 22:49:05.737328 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:05.737304    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pdpx_b8dfe233-77be-496f-9683-9b61adea1438/kube-rbac-proxy/0.log"
Jul 16 22:49:05.760707 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:05.760626    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pdpx_b8dfe233-77be-496f-9683-9b61adea1438/init-textfile/0.log"
Jul 16 22:49:06.050044 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.049916    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/prometheus/0.log"
Jul 16 22:49:06.073248 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.073200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/config-reloader/0.log"
Jul 16 22:49:06.106346 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.106256    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/thanos-sidecar/0.log"
Jul 16 22:49:06.130303 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.130267    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/kube-rbac-proxy-web/0.log"
Jul 16 22:49:06.153218 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.153189    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/kube-rbac-proxy/0.log"
Jul 16 22:49:06.176737 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.176704    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/kube-rbac-proxy-thanos/0.log"
Jul 16 22:49:06.207607 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.207578    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc710204-7ef3-4629-ac3e-714d5962cc56/init-config-reloader/0.log"
Jul 16 22:49:06.237324 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.237290    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-zv6dz_6ea83b32-dcb9-4d65-a387-8ea4c6f878cf/prometheus-operator/0.log"
Jul 16 22:49:06.259905 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.259882    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-zv6dz_6ea83b32-dcb9-4d65-a387-8ea4c6f878cf/kube-rbac-proxy/0.log"
Jul 16 22:49:06.882533 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.882485    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"]
Jul 16 22:49:06.888809 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.888780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:06.902060 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.902031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"]
Jul 16 22:49:06.981828 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.981773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmh5\" (UniqueName: \"kubernetes.io/projected/d4f8c689-2b94-4aed-94f0-572b68541a6d-kube-api-access-qtmh5\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:06.982039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.981872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-podres\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:06.982039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.981911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-sys\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:06.982039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.981940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-proc\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:06.982039 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:06.981992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-lib-modules\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083071 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-sys\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083266 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-proc\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083266 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-lib-modules\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083266 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmh5\" (UniqueName: \"kubernetes.io/projected/d4f8c689-2b94-4aed-94f0-572b68541a6d-kube-api-access-qtmh5\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083266 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-podres\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083266 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-proc\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-sys\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-lib-modules\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.083510 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.083358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4f8c689-2b94-4aed-94f0-572b68541a6d-podres\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.093381 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.093355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmh5\" (UniqueName: \"kubernetes.io/projected/d4f8c689-2b94-4aed-94f0-572b68541a6d-kube-api-access-qtmh5\") pod \"perf-node-gather-daemonset-7cbwz\" (UID: \"d4f8c689-2b94-4aed-94f0-572b68541a6d\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.203904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.203872    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:07.378893 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.378762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"]
Jul 16 22:49:07.475629 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:07.475500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz" event={"ID":"d4f8c689-2b94-4aed-94f0-572b68541a6d","Type":"ContainerStarted","Data":"9e656d49e09621990343de90ef3a132075e3f99811a8b23d0a9a120f5e8c7047"}
Jul 16 22:49:08.483444 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:08.483391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz" event={"ID":"d4f8c689-2b94-4aed-94f0-572b68541a6d","Type":"ContainerStarted","Data":"bf8e39ce6e5bc4768242d1df00355b72af19376efb25743168effe80a020588a"}
Jul 16 22:49:08.483904 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:08.483617    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:08.501443 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:08.501392    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz" podStartSLOduration=2.501376785 podStartE2EDuration="2.501376785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:49:08.500409549 +0000 UTC m=+3731.282307400" watchObservedRunningTime="2026-07-16 22:49:08.501376785 +0000 UTC m=+3731.283274634"
Jul 16 22:49:09.325188 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:09.325156    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-9cldx_7cda6ed7-0b12-445e-8e15-4b378a623acf/volume-data-source-validator/0.log"
Jul 16 22:49:10.418706 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:10.418680    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hqdk7_607c5dc3-2461-4b05-a37a-413b4a681f38/dns/0.log"
Jul 16 22:49:10.441156 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:10.441089    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hqdk7_607c5dc3-2461-4b05-a37a-413b4a681f38/kube-rbac-proxy/0.log"
Jul 16 22:49:10.512542 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:10.512514    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-98t4h_4e344137-20c3-47a9-97c0-a800bf47e475/dns-node-resolver/0.log"
Jul 16 22:49:11.178593 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:11.178541    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6q4k4_db43cfec-8a7b-4fc6-8370-c6f8dae02a69/node-ca/0.log"
Jul 16 22:49:12.594708 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:12.594671    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-76tr8_98626467-5488-41b5-a3f5-498667623a37/kube-auth-proxy/0.log"
Jul 16 22:49:12.783238 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:12.783209    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-r8vkg_a9cf7475-5dcb-40af-8636-1882c45278f3/payload-pre-processing/0.log"
Jul 16 22:49:13.600403 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:13.600373    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ds5ls_4dbdd52d-ece0-471e-8645-b79df023e5ac/serve-healthcheck-canary/0.log"
Jul 16 22:49:14.224370 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:14.224342    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2r7n_76683669-4321-4952-9731-007a95935967/kube-rbac-proxy/0.log"
Jul 16 22:49:14.247044 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:14.247020    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2r7n_76683669-4321-4952-9731-007a95935967/exporter/0.log"
Jul 16 22:49:14.275987 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:14.275954    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2r7n_76683669-4321-4952-9731-007a95935967/extractor/0.log"
Jul 16 22:49:14.501008 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:14.500933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-7cbwz"
Jul 16 22:49:17.178000 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:17.177921    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-95466d4fd-mdsdq_c5f395d0-76cf-47bf-a1c4-1e15a9ed061c/manager/0.log"
Jul 16 22:49:17.387855 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:17.387829    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-76tr8_98626467-5488-41b5-a3f5-498667623a37/kube-auth-proxy/0.log"
Jul 16 22:49:17.495928 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:17.495849    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-r8vkg_a9cf7475-5dcb-40af-8636-1882c45278f3/payload-pre-processing/0.log"
Jul 16 22:49:17.538126 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:17.538090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-vt7sp_9debcaac-f977-4b23-840d-81dd2ab761d8/manager/0.log"
Jul 16 22:49:17.561093 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:17.561067    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-vt7sp_9debcaac-f977-4b23-840d-81dd2ab761d8/copy-manifests/0.log"
Jul 16 22:49:19.006824 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:19.006798    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c6c96995f-xmgpv_554f6cf4-30e2-434b-98bb-bc9ef8393691/manager/0.log"
Jul 16 22:49:19.070513 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:19.070484    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-76tr8_98626467-5488-41b5-a3f5-498667623a37/kube-auth-proxy/0.log"
Jul 16 22:49:19.171182 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:19.171156    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-r8vkg_a9cf7475-5dcb-40af-8636-1882c45278f3/payload-pre-processing/0.log"
Jul 16 22:49:24.246095 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:24.246063    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sssnm_d5ded2d9-965e-4a28-a63f-7fb80d6a0960/migrator/0.log"
Jul 16 22:49:24.270124 ip-10-0-139-156 kubenswrapper[2562]: I0716 22:49:24.270096    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sssnm_d5ded2d9-965e-4a28-a63f-7fb80d6a0960/graceful-termination/0.log"