Jul 17 21:27:15.958903 ip-10-0-139-16 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 21:27:16.401560 ip-10-0-139-16 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:27:16.401560 ip-10-0-139-16 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 21:27:16.401560 ip-10-0-139-16 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:27:16.401560 ip-10-0-139-16 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 21:27:16.401560 ip-10-0-139-16 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:27:16.403923 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.403809    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 21:27:16.410114 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410088    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:27:16.410114 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410112    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:27:16.410114 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410116    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:27:16.410114 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410119    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:27:16.410114 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410121    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:27:16.410114 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410125    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410127    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410130    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410133    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410135    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410140    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410142    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410145    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410149    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410153    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410157    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410160    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410163    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410166    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410169    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410172    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410175    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410178    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410181    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:27:16.410349 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410185    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410189    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410192    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410195    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410198    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410200    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410203    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410205    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410208    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410211    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410214    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410216    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410219    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410221    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410224    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410227    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410230    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410232    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410237    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410241    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:27:16.410854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410243    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410246    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410249    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410251    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410254    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410256    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410259    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410261    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410264    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410266    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410269    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410271    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410274    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410276    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410280    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410283    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410285    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410288    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410291    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410293    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:27:16.411343 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410296    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410299    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410302    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410305    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410307    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410310    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410312    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410315    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410317    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410320    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410322    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410325    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410328    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410331    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410333    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410335    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410339    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410342    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410344    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410347    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:27:16.411854 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410350    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410352    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410355    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410358    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410360    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410363    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410366    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410368    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410371    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410374    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410376    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410379    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410381    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410384    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.410386    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411257    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411267    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411271    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411274    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:27:16.412336 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411278    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411281    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411284    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411287    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411290    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411293    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411295    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411298    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411301    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411304    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411307    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411309    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411312    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411314    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411317    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411319    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411322    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411325    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411327    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411330    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:27:16.412800 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411334    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411336    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411339    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411341    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411344    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411348    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411352    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411355    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411358    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411361    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411363    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411365    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411368    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411371    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411373    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411375    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411378    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411380    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411383    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:27:16.413290 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411385    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411388    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411390    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411392    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411395    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411397    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411400    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411403    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411405    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411408    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411410    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411413    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411416    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411419    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411423    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411426    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411430    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411433    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411436    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411439    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:27:16.413954 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411442    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411445    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411448    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411451    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411454    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411457    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411460    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411463    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411466    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411468    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411471    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411473    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411476    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411478    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411481    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411483    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411486    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411488    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411491    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411494    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:27:16.414765 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411496    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411500    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411502    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411504    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411507    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411510    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411513    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411515    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411518    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411520    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411523    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411525    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411528    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411531    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411533    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.411536    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413355    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413365    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413369    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413372    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:27:16.415359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413375    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413378    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413381    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413384    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413387    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413394    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413397    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413400    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413402    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413405    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413408    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413411    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413413    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413416    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413419    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413422    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413424    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413427    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413433    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413435    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:27:16.415868 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413438    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413441    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413444    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413447    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413449    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413452    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413455    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413458    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413461    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413464    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413470    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413474    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413478    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413482    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413485    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413488    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413491    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413494    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413497    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:27:16.416359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413500    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413502    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413505    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413508    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413515    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413519    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.413523    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415677    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415686    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415689    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415694    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415698    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415701    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415704    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415707    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415710    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415718    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415723    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415727    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415731    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:27:16.416881 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415733    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415736    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415741    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415744    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415747    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415750    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415754    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415756    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415762    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415765    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415768    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415770    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415774    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415777    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415779    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415782    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415785    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415788    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415790    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:27:16.417357 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415793    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415798    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415804    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415808    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415811    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415814    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415817    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415821    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415824    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415827    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415830    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415833    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415836    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415842    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415845    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415848    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.415853    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.415976    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.415984    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.415993    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 21:27:16.417924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.415998    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416002    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416006    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416014    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416019    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416023    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416026    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416030    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416034    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416037    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416041    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416046    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416050    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416053    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416056    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416060    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416063    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416067    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416070    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416074    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416080    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416083    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416086    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 21:27:16.418442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416090    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416093    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416097    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416100    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416103    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416109    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416115    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416118    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416122    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416125    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416128    2563 flags.go:64] FLAG: --enable-server="true"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416131    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416138    2563 flags.go:64] FLAG: --event-burst="100"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416141    2563 flags.go:64] FLAG: --event-qps="50"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416146    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416150    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416153    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416158    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416161    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416164    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416167    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416171    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416177    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416180    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416184    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 21:27:16.419007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416187    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416190    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416193    2563 flags.go:64] FLAG: --feature-gates=""
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416197    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416201    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416206    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416210    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416213    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416217    2563 flags.go:64] FLAG: --help="false"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416220    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416253    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416257    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416261    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416283    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416288    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416386    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416391    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416395    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416398    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416401    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416404    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416407    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416410    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416413    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 21:27:16.419644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416416    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416419    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416422    2563 flags.go:64] FLAG: --lock-file=""
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416425    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416428    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416431    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416437    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416440    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416442    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416446    2563 flags.go:64] FLAG: --logging-format="text"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416449    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416452    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416455    2563 flags.go:64] FLAG: --manifest-url=""
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416458    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416463    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416466    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416471    2563 flags.go:64] FLAG: --max-pods="110"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416474    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416477    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416480    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416483    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416486    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416489    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416492    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416501    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 21:27:16.420219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416505    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416508    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416511    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416514    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416520    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416523    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416526    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416529    2563 flags.go:64] FLAG: --port="10250"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416532    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416535    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f024d2421aa59241"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416538    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416541    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416544    2563 flags.go:64] FLAG: --register-node="true"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416547    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416550    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416553    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416557    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416559    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416563    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416566    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416569    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416572    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416575    2563 flags.go:64] FLAG: --runonce="false"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416578    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416582    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 21:27:16.420861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416585    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416587    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416590    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416606    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416610    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416614    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416620    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416623    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416626    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416629    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416632    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416635    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416638    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416644    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416647    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416650    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416654    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416657    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416660    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416663    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416666    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416669    2563 flags.go:64] FLAG: --v="2"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416673    2563 flags.go:64] FLAG: --version="false"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416677    2563 flags.go:64] FLAG: --vmodule=""
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416681    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 21:27:16.421491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.416685    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416824    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416828    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416831    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416834    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416837    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416840    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416843    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416845    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416848    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416850    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416853    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416855    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416861    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416864    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416867    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416869    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416872    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416875    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416877    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:27:16.422107 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416881    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416885    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416888    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416890    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416893    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416896    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416899    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416901    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416904    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416907    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416909    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416912    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416915    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416918    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416920    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416923    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416925    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416928    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416930    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416933    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:27:16.422580 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416935    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416938    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416940    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416944    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416948    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416952    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416955    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416958    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416961    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416963    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416966    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416969    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416971    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416973    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416976    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416978    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416981    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416984    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416987    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:27:16.423083 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416989    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416992    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416994    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416997    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.416999    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417002    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417005    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417007    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417010    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417012    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417015    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417018    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417020    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417023    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417025    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417028    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417030    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417033    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417038    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:27:16.423554 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417043    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417047    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417050    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417053    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417055    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417058    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417061    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417064    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417066    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417069    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417071    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417073    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417076    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417078    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417081    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417084    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417086    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417088    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417091    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417094    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:27:16.424031 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417097    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:27:16.424506 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.417099    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:27:16.424506 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.417990    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:27:16.426316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.426296    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 21:27:16.426316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.426317    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 21:27:16.426376 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.426337    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 21:27:16.426376 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.426344    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426400    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426405    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426409    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426413    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426416    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426419    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426421    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426424    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426427    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:27:16.426426 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426430    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426432    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426435    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426437    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426440    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426443    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426445    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426448    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426450    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426453    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426455    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426457    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426460    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426464    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426468    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426471    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426474    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426477    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426480    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426482    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:27:16.426745 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426486    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426488    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426491    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426493    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426496    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426498    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426501    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426503    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426506    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426508    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426510    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426513    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426515    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426518    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426521    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426524    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426526    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426529    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426532    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426534    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:27:16.427244 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426537    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426539    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426542    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426544    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426548    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426551    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426554    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426557    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426560    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426562    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426565    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426569    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426572    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426576    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426578    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426581    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426583    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426586    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426588    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:27:16.427758 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426591    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426608    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426633    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426639    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426642    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426644    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426647    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426650    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426653    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426655    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426658    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426661    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426663    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426666    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426669    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426671    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426673    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426676    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426678    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426681    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:27:16.428219 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426683    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426686    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426688    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426691    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426693    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426696    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426698    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426702    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426704    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426707    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426709    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.426715    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426831    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426836    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426839    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:27:16.428753 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426842    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426845    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426847    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426850    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426852    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426855    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426857    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426860    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426862    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426865    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426868    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426870    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426872    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426875    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426877    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426880    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426882    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426885    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426887    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426890    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:27:16.429118 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426892    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426895    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426897    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426900    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426902    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426905    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426908    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426910    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426913    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426917    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426920    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426924    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426926    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426929    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426932    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426934    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426937    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426939    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426942    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426944    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:27:16.429611 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426947    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426949    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426952    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426954    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426957    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426959    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426961    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426964    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426966    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426969    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426971    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426974    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426976    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426979    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426981    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426984    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426987    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426991    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426994    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426996    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:27:16.430078 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.426999    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427001    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427004    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427007    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427009    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427012    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427015    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427018    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427020    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427022    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427025    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427027    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427030    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427032    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427034    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427037    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427039    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427042    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427044    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:27:16.430546 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427047    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427049    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427052    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427054    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427057    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427059    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427061    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427064    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427068    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427071    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427074    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427077    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427080    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427083    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427086    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427089    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:27:16.431020 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:16.427092    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:27:16.431394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.427096    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:27:16.431394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.427239    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 21:27:16.431394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.429614    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 21:27:16.431394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.431188    2563 server.go:1019] "Starting client certificate rotation"
Jul 17 21:27:16.431394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.431294    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:27:16.432183 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.432170    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:27:16.434048 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.434035    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:27:16.435166 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.435149    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:27:16.446123 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.446108    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 17 21:27:16.448773 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.448751    2563 log.go:25] "Validated CRI v1 image API"
Jul 17 21:27:16.450704 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.450684    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 21:27:16.456531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.456512    2563 fs.go:135] Filesystem UUIDs: map[139e9a8b-431b-4e8f-ab56-d4dd64167c4e:/dev/nvme0n1p4 32910e68-38fe-42d8-8a8c-505ccb7ff6ce:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 17 21:27:16.456621 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.456531    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 21:27:16.462084 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.461948    2563 manager.go:216] Machine: {Timestamp:2026-07-17 21:27:16.460488914 +0000 UTC m=+0.372729197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096770 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eea17ef1a3edf955359281053cca8 SystemUUID:ec2eea17-ef1a-3edf-9553-59281053cca8 BootID:0ce3fbf2-4fc8-4b6d-88a5-12282be4a529 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:1c:9d:95:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:1c:9d:95:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:49:d3:db:2e:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 21:27:16.462084 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.462078    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 21:27:16.462186 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.462160    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 21:27:16.464150 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.464133    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:27:16.466519 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.466494    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 21:27:16.466673 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.466522    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 21:27:16.467747 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.467737    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 21:27:16.467781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.467750    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 21:27:16.467781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.467758    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:27:16.468290 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.468281    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:27:16.469727 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.469717    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 21:27:16.472789 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472777    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:27:16.472906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472897    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 21:27:16.472942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472928    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 21:27:16.472942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472939    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 21:27:16.472998 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472955    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 21:27:16.472998 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472965    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 17 21:27:16.472998 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.472975    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 21:27:16.473349 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.473336    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:27:16.473390 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.473362    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:27:16.475838 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.475816    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 21:27:16.478813 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.478799    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 21:27:16.478874 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.478829    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 21:27:16.480054 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480041    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480059    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480065    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480070    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480079    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480085    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480093    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480098    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480106    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480111    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 21:27:16.480115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480120    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 21:27:16.480404 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480125    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 21:27:16.481002 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.480991    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 21:27:16.481002 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.481002    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 21:27:16.484200 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.484181    2563 server.go:1268] "Started kubelet"
Jul 17 21:27:16.484390 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.484292    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 21:27:16.484490 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.484362    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 21:27:16.484490 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.484431    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 21:27:16.484633 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.484570    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 21:27:16.484742 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.484723    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 21:27:16.484846 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.484829    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 21:27:16.485068 ip-10-0-139-16 systemd[1]: Started Kubernetes Kubelet.
Jul 17 21:27:16.488028 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.488007    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 21:27:16.489517 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.489499    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 21:27:16.492452 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.492428    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 21:27:16.496255 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.495104    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-16.ec2.internal.18c330e4c4f8a043  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-16.ec2.internal,UID:ip-10-0-139-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-16.ec2.internal,},FirstTimestamp:2026-07-17 21:27:16.484153411 +0000 UTC m=+0.396393691,LastTimestamp:2026-07-17 21:27:16.484153411 +0000 UTC m=+0.396393691,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-16.ec2.internal,}"
Jul 17 21:27:16.496843 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.496823    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 21:27:16.496843 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.496836    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 21:27:16.497094 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.497079    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:16.497720 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.497706    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 21:27:16.497854 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.497758    2563 factory.go:55] Registering systemd factory
Jul 17 21:27:16.497963 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.497858    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 17 21:27:16.498115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498101    2563 factory.go:153] Registering CRI-O factory
Jul 17 21:27:16.498182 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498118    2563 factory.go:223] Registration of the crio container factory successfully
Jul 17 21:27:16.498182 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498139    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 21:27:16.498182 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498155    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 21:27:16.498319 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498185    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 21:27:16.498319 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498210    2563 factory.go:103] Registering Raw factory
Jul 17 21:27:16.498319 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498226    2563 manager.go:1196] Started watching for new ooms in manager
Jul 17 21:27:16.498319 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498273    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 21:27:16.498319 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498286    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 21:27:16.498938 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.498921    2563 manager.go:349] Starting recovery of all containers
Jul 17 21:27:16.500309 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.500284    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 21:27:16.500973 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.500558    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 21:27:16.509548 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.509531    2563 manager.go:354] Recovery completed
Jul 17 21:27:16.513819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.513802    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.514526 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.514510    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v2vzw"
Jul 17 21:27:16.516040 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.516021    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.516127 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.516054    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.516127 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.516076    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.517080 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.517063    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 21:27:16.517080 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.517076    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.517236 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.517078    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 21:27:16.517236 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.517174    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:27:16.517954 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.517873    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-16.ec2.internal.18c330e4c6df2de7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-16.ec2.internal,UID:ip-10-0-139-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-16.ec2.internal,},FirstTimestamp:2026-07-17 21:27:16.516040167 +0000 UTC m=+0.428280453,LastTimestamp:2026-07-17 21:27:16.516040167 +0000 UTC m=+0.428280453,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-16.ec2.internal,}"
Jul 17 21:27:16.518122 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.518107    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.518209 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.518142    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.518209 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.518158    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.518590 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.518577    2563 policy_none.go:49] "None policy: Start"
Jul 17 21:27:16.518650 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.518613    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 21:27:16.518650 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.518625    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 21:27:16.519672 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.519661    2563 policy_none.go:47] "Start"
Jul 17 21:27:16.521776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.521667    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v2vzw"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561174    2563 manager.go:337] "Starting Device Plugin manager"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.561202    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561213    2563 server.go:87] "Starting device plugin registration server"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561476    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561490    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561612    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561705    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.561715    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.562195    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 21:27:16.562786 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.562238    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:16.594657 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.594628    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 21:27:16.595966 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.595952    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 21:27:16.596044 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.595986    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 21:27:16.596044 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.596003    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 21:27:16.596044 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.596037    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 21:27:16.598937 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.598922    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:27:16.662043 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.661973    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.662892 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.662876    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.662942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.662905    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.662942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.662916    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.662942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.662939    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.671535 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.671521    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.671631 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.671542    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-16.ec2.internal\": node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:16.696844 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.696818    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"]
Jul 17 21:27:16.696926 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.696883    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.697704 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.697687    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.697799 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.697720    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.697799 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.697736    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.698853 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.698831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab437d1b1f10a17975f1fc588a4bc307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"ab437d1b1f10a17975f1fc588a4bc307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.698990 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.698857    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.698990 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.698858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab437d1b1f10a17975f1fc588a4bc307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"ab437d1b1f10a17975f1fc588a4bc307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.698990 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.698966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.699143 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.698996    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.699560 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.699544    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.699679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.699562    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.699679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.699573    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.699679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.699588    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.699679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.699589    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.699679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.699630    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.700759 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.700744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.700848 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.700769    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:27:16.701515 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.701499    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:27:16.701585 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.701525    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:27:16.701585 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.701538    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:27:16.708761 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.708745    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:16.716185 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.716162    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-16.ec2.internal\" not found" node="ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.720264 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.720242    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-16.ec2.internal\" not found" node="ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.799313 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.799288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab437d1b1f10a17975f1fc588a4bc307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"ab437d1b1f10a17975f1fc588a4bc307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.799313 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.799316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab437d1b1f10a17975f1fc588a4bc307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"ab437d1b1f10a17975f1fc588a4bc307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.799487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.799336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a28b61f94e609f57bdc8bf0691328b0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-16.ec2.internal\" (UID: \"1a28b61f94e609f57bdc8bf0691328b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.799487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.799384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab437d1b1f10a17975f1fc588a4bc307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"ab437d1b1f10a17975f1fc588a4bc307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.799487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.799389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab437d1b1f10a17975f1fc588a4bc307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"ab437d1b1f10a17975f1fc588a4bc307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.809436 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.809403    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:16.899911 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.899886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a28b61f94e609f57bdc8bf0691328b0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-16.ec2.internal\" (UID: \"1a28b61f94e609f57bdc8bf0691328b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.900052 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:16.899944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a28b61f94e609f57bdc8bf0691328b0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-16.ec2.internal\" (UID: \"1a28b61f94e609f57bdc8bf0691328b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:16.910025 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:16.910003    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.010905 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:17.010837    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.020032 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.020005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:17.022328 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.022311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:17.111484 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:17.111445    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.212008 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:17.211976    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.312545 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:17.312471    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.413026 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:17.412998    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.430174 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.430151    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 21:27:17.430332 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.430313    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:27:17.475694 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.475670    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:27:17.513968 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:17.513941    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found"
Jul 17 21:27:17.526113 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.526082    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:22:17 +0000 UTC" deadline="2028-02-07 06:47:41.452878536 +0000 UTC"
Jul 17 21:27:17.526113 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.526108    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13665h20m23.92677348s"
Jul 17 21:27:17.551486 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.551465    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:27:17.597665 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.597592    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:17.611273 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.611255    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:27:17.612562 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.612549    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"
Jul 17 21:27:17.620827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.620812    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:27:17.743867 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:17.743838    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a28b61f94e609f57bdc8bf0691328b0.slice/crio-a354ad923dc28f0856ab1f1c7cab4fb013f8cafb7e6a3841b6cb7dc5daacfbdf WatchSource:0}: Error finding container a354ad923dc28f0856ab1f1c7cab4fb013f8cafb7e6a3841b6cb7dc5daacfbdf: Status 404 returned error can't find the container with id a354ad923dc28f0856ab1f1c7cab4fb013f8cafb7e6a3841b6cb7dc5daacfbdf
Jul 17 21:27:17.749186 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:17.749169    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:27:17.762895 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:17.762865    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab437d1b1f10a17975f1fc588a4bc307.slice/crio-61940a3e5d3ab6f5dec5ae72ee28040b801ebd07c2229fcf89bd6444609884b8 WatchSource:0}: Error finding container 61940a3e5d3ab6f5dec5ae72ee28040b801ebd07c2229fcf89bd6444609884b8: Status 404 returned error can't find the container with id 61940a3e5d3ab6f5dec5ae72ee28040b801ebd07c2229fcf89bd6444609884b8
Jul 17 21:27:18.075564 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.075484    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:27:18.473921 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.473840    2563 apiserver.go:52] "Watching apiserver"
Jul 17 21:27:18.484012 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.483980    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 21:27:18.485041 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.485013    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jqdbf","kube-system/konnectivity-agent-sqnqz","openshift-image-registry/node-ca-l9c7k","openshift-multus/multus-swkc4","openshift-network-diagnostics/network-check-target-4nctl","openshift-network-operator/iptables-alerter-zm8fg","openshift-ovn-kubernetes/ovnkube-node-dh88b","kube-system/global-pull-secret-syncer-dz78x","kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb","openshift-cluster-node-tuning-operator/tuned-lkrjc","openshift-dns/node-resolver-97sf8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal","openshift-multus/multus-additional-cni-plugins-27mht"]
Jul 17 21:27:18.486735 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.486710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.488256 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.487869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.489363 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.489333    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 21:27:18.489477 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.489336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.489543 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.489488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 21:27:18.489543 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.489536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnf27\""
Jul 17 21:27:18.489899 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.489881    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 21:27:18.490069 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 21:27:18.490194 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.490273 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.490332 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.490476 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2br9\""
Jul 17 21:27:18.490584 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.490780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.490748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.491883 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.491863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:18.491996 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.491951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:18.492329 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.492308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-762kt\""
Jul 17 21:27:18.492539 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.492522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 21:27:18.492725 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.492707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.493794 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.493067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 21:27:18.493794 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.493404    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.493794 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.493595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 21:27:18.494011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.493866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.494011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.493881    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hpncw\""
Jul 17 21:27:18.494011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.493870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.494366 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.494270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.495497 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.495469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.496827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.496189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vcpd\""
Jul 17 21:27:18.496827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.496461    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.496827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.496490    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.496827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.496657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.496827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.496674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 21:27:18.496827 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.496750    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:18.497344 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.497324    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 21:27:18.497573 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.497544    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 21:27:18.498640 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.497987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 21:27:18.498640 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.498149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.498810 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.498740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c9rvn\""
Jul 17 21:27:18.499617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.499584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.500345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.500327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pl4bj\""
Jul 17 21:27:18.500900 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.500645    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 21:27:18.500900 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.500706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 21:27:18.501342 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.501320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:18.501442 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.501397    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:18.501773 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.501749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wpfx\""
Jul 17 21:27:18.502043 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.502015    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.502157 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.502105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 21:27:18.502157 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.502131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.502852 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.502831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.505028 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.505009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hmqpp\""
Jul 17 21:27:18.508879 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.508857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 21:27:18.508989 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.508955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 21:27:18.510849 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-iptables-alerter-script\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.510957 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-os-release\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.510957 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.510957 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-var-lib-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.510957 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-systemd\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.510957 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovnkube-script-lib\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.510957 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-lib-modules\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-host\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90e3694e-96d4-4374-b47e-b666fc83db50-cni-binary-copy\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.510996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-socket-dir-parent\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-cni-multus\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-ovn\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-cni-bin\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-systemd-units\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-run-netns\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-log-socket\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-modprobe-d\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-systemd\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-run\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-sys\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-host\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511242 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-kubelet\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-hostroot\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rwj\" (UniqueName: \"kubernetes.io/projected/40814c12-245a-4369-bfc8-4ea9cd93e60f-kube-api-access-f6rwj\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-kubelet-config\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-dbus\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-kubernetes\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-serviceca\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-host-slash\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-kubelet\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnt9\" (UniqueName: \"kubernetes.io/projected/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-kube-api-access-drnt9\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-var-lib-kubelet\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-conf-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-etc-kubernetes\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgsx\" (UniqueName: \"kubernetes.io/projected/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-kube-api-access-2xgsx\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-system-cni-dir\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.511901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-etc-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-run-ovn-kubernetes\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-cni-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-k8s-cni-cncf-io\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysctl-d\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-cnibin\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-node-log\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90e3694e-96d4-4374-b47e-b666fc83db50-multus-daemon-config\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-cnibin\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-cni-netd\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-netns\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76b0098e-e168-452f-a057-d6488676bb28-agent-certs\") pod \"konnectivity-agent-sqnqz\" (UID: \"76b0098e-e168-452f-a057-d6488676bb28\") " pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76b0098e-e168-452f-a057-d6488676bb28-konnectivity-ca\") pod \"konnectivity-agent-sqnqz\" (UID: \"76b0098e-e168-452f-a057-d6488676bb28\") " pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysctl-conf\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxh7z\" (UniqueName: \"kubernetes.io/projected/adad1485-282e-484c-ba4b-ff8bb8ffd410-kube-api-access-hxh7z\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.512493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-system-cni-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mtkd\" (UniqueName: \"kubernetes.io/projected/90e3694e-96d4-4374-b47e-b666fc83db50-kube-api-access-4mtkd\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-cni-binary-copy\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-env-overrides\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adad1485-282e-484c-ba4b-ff8bb8ffd410-tmp\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7964\" (UniqueName: \"kubernetes.io/projected/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-kube-api-access-v7964\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-os-release\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-cni-bin\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.511989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-multus-certs\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysconfig\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-slash\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovnkube-config\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovn-node-metrics-cert\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-tuned\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.513335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.512775    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:27:18.551534 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.551507    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-76vzz"
Jul 17 21:27:18.560304 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.560284    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-76vzz"
Jul 17 21:27:18.600324 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.600237    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 21:27:18.603955 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.603905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"ab437d1b1f10a17975f1fc588a4bc307","Type":"ContainerStarted","Data":"61940a3e5d3ab6f5dec5ae72ee28040b801ebd07c2229fcf89bd6444609884b8"}
Jul 17 21:27:18.605830 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.605802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" event={"ID":"1a28b61f94e609f57bdc8bf0691328b0","Type":"ContainerStarted","Data":"a354ad923dc28f0856ab1f1c7cab4fb013f8cafb7e6a3841b6cb7dc5daacfbdf"}
Jul 17 21:27:18.613130 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a05c180a-04d3-4fa2-904d-034acaf5ada9-hosts-file\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.613247 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-systemd-units\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.613247 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-run-netns\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.613247 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-log-socket\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.613247 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-modprobe-d\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613247 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-systemd-units\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-systemd\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-run-netns\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-run\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-sys\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-host\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-kubelet\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.613453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-modprobe-d\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-hostroot\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-run\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rwj\" (UniqueName: \"kubernetes.io/projected/40814c12-245a-4369-bfc8-4ea9cd93e60f-kube-api-access-f6rwj\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-systemd\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-kubelet-config\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-log-socket\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-dbus\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-registration-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-dbus\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-kubernetes\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-serviceca\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-kubernetes\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.613755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-host-slash\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-hostroot\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-host-slash\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-kubelet\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnt9\" (UniqueName: \"kubernetes.io/projected/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-kube-api-access-drnt9\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-var-lib-kubelet\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-conf-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-etc-kubernetes\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.613988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgsx\" (UniqueName: \"kubernetes.io/projected/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-kube-api-access-2xgsx\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-system-cni-dir\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-etc-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-run-ovn-kubernetes\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-cni-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-kubelet-config\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-k8s-cni-cncf-io\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-device-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.614263 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysctl-d\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-cnibin\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-socket-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9glf\" (UniqueName: \"kubernetes.io/projected/a05c180a-04d3-4fa2-904d-034acaf5ada9-kube-api-access-r9glf\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-node-log\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90e3694e-96d4-4374-b47e-b666fc83db50-multus-daemon-config\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-cnibin\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-cni-netd\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-netns\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76b0098e-e168-452f-a057-d6488676bb28-agent-certs\") pod \"konnectivity-agent-sqnqz\" (UID: \"76b0098e-e168-452f-a057-d6488676bb28\") " pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76b0098e-e168-452f-a057-d6488676bb28-konnectivity-ca\") pod \"konnectivity-agent-sqnqz\" (UID: \"76b0098e-e168-452f-a057-d6488676bb28\") " pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysctl-conf\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-serviceca\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.614906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxh7z\" (UniqueName: \"kubernetes.io/projected/adad1485-282e-484c-ba4b-ff8bb8ffd410-kube-api-access-hxh7z\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-system-cni-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbx59\" (UniqueName: \"kubernetes.io/projected/68e20af5-a9a5-464f-81d1-67a8c17db4cf-kube-api-access-mbx59\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-sys\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mtkd\" (UniqueName: \"kubernetes.io/projected/90e3694e-96d4-4374-b47e-b666fc83db50-kube-api-access-4mtkd\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-cni-binary-copy\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-host\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-env-overrides\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-kubelet\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-kubelet\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adad1485-282e-484c-ba4b-ff8bb8ffd410-tmp\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7964\" (UniqueName: \"kubernetes.io/projected/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-kube-api-access-v7964\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-os-release\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-cni-bin\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-cnibin\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-multus-certs\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysconfig\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.615606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-node-log\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-slash\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnw9\" (UniqueName: \"kubernetes.io/projected/b585eadd-75f4-4d2c-828f-1a21849d59dd-kube-api-access-9gnw9\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c180a-04d3-4fa2-904d-034acaf5ada9-tmp-dir\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovnkube-config\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovn-node-metrics-cert\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-tuned\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-iptables-alerter-script\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-os-release\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-var-lib-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-sys-fs\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-systemd\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovnkube-script-lib\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-lib-modules\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.616395 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-host\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90e3694e-96d4-4374-b47e-b666fc83db50-cni-binary-copy\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-socket-dir-parent\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-cni-multus\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-ovn\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90e3694e-96d4-4374-b47e-b666fc83db50-multus-daemon-config\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-cni-bin\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-var-lib-kubelet\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.615985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-conf-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-etc-kubernetes\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-system-cni-dir\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-etc-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-run-ovn-kubernetes\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-cni-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-k8s-cni-cncf-io\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-cni-binary-copy\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.617154 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.616535    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.616625    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:19.116591706 +0000 UTC m=+3.028831991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.616736    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-env-overrides\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovnkube-script-lib\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-os-release\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.614958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysctl-d\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-cni-bin\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-multus-certs\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysconfig\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-slash\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovnkube-config\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.617742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-var-lib-cni-multus\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-lib-modules\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-host\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-multus-socket-dir-parent\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-ovn\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90e3694e-96d4-4374-b47e-b666fc83db50-cni-binary-copy\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-cni-bin\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.618778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-cnibin\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.619142 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.619142 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.618846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-var-lib-openvswitch\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.619142 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-run-systemd\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.619258 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40814c12-245a-4369-bfc8-4ea9cd93e60f-os-release\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.619340 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-sysctl-conf\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.619406 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-iptables-alerter-script\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.619462 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-host-cni-netd\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.619882 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40814c12-245a-4369-bfc8-4ea9cd93e60f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.619974 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76b0098e-e168-452f-a057-d6488676bb28-konnectivity-ca\") pod \"konnectivity-agent-sqnqz\" (UID: \"76b0098e-e168-452f-a057-d6488676bb28\") " pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.619974 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-system-cni-dir\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.620086 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.619973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90e3694e-96d4-4374-b47e-b666fc83db50-host-run-netns\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.620832 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.620769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adad1485-282e-484c-ba4b-ff8bb8ffd410-tmp\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.624810 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.624785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-ovn-node-metrics-cert\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.624903 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.624795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adad1485-282e-484c-ba4b-ff8bb8ffd410-etc-tuned\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.626746 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.626703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76b0098e-e168-452f-a057-d6488676bb28-agent-certs\") pod \"konnectivity-agent-sqnqz\" (UID: \"76b0098e-e168-452f-a057-d6488676bb28\") " pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.633087 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.632884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rwj\" (UniqueName: \"kubernetes.io/projected/40814c12-245a-4369-bfc8-4ea9cd93e60f-kube-api-access-f6rwj\") pod \"multus-additional-cni-plugins-27mht\" (UID: \"40814c12-245a-4369-bfc8-4ea9cd93e60f\") " pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.633804 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.633730    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:18.633804 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.633759    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:18.633804 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.633786    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:18.633804 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.633867    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:27:19.133840608 +0000 UTC m=+3.046080896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:18.638270 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.638212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mtkd\" (UniqueName: \"kubernetes.io/projected/90e3694e-96d4-4374-b47e-b666fc83db50-kube-api-access-4mtkd\") pod \"multus-swkc4\" (UID: \"90e3694e-96d4-4374-b47e-b666fc83db50\") " pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.638446 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.638327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7964\" (UniqueName: \"kubernetes.io/projected/4fdc8bd8-1135-46ad-a9e9-4e06928fc07e-kube-api-access-v7964\") pod \"node-ca-l9c7k\" (UID: \"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e\") " pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.639047 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.638988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxh7z\" (UniqueName: \"kubernetes.io/projected/adad1485-282e-484c-ba4b-ff8bb8ffd410-kube-api-access-hxh7z\") pod \"tuned-lkrjc\" (UID: \"adad1485-282e-484c-ba4b-ff8bb8ffd410\") " pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.639425 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.639379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnt9\" (UniqueName: \"kubernetes.io/projected/80dbcaea-ab7d-4abd-9894-4db5d4e15dba-kube-api-access-drnt9\") pod \"ovnkube-node-dh88b\" (UID: \"80dbcaea-ab7d-4abd-9894-4db5d4e15dba\") " pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.639932 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.639894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgsx\" (UniqueName: \"kubernetes.io/projected/d2887a37-5f1f-42bd-b9ad-36a419bf1cdd-kube-api-access-2xgsx\") pod \"iptables-alerter-zm8fg\" (UID: \"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd\") " pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.716004 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.715965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-device-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716191 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-socket-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716191 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9glf\" (UniqueName: \"kubernetes.io/projected/a05c180a-04d3-4fa2-904d-034acaf5ada9-kube-api-access-r9glf\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.716191 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716191 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716191 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:18.716191 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbx59\" (UniqueName: \"kubernetes.io/projected/68e20af5-a9a5-464f-81d1-67a8c17db4cf-kube-api-access-mbx59\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnw9\" (UniqueName: \"kubernetes.io/projected/b585eadd-75f4-4d2c-828f-1a21849d59dd-kube-api-access-9gnw9\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c180a-04d3-4fa2-904d-034acaf5ada9-tmp-dir\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-sys-fs\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a05c180a-04d3-4fa2-904d-034acaf5ada9-hosts-file\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-registration-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-registration-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-device-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716900 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-socket-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.716959 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.716928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.718951 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.717034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.718951 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.717101    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:18.718951 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:18.717155    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:27:19.217140136 +0000 UTC m=+3.129380406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:18.718951 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.717659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c180a-04d3-4fa2-904d-034acaf5ada9-tmp-dir\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.718951 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.717743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b585eadd-75f4-4d2c-828f-1a21849d59dd-sys-fs\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.718951 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.717790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a05c180a-04d3-4fa2-904d-034acaf5ada9-hosts-file\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.729986 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.729925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbx59\" (UniqueName: \"kubernetes.io/projected/68e20af5-a9a5-464f-81d1-67a8c17db4cf-kube-api-access-mbx59\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:18.733629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.733599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9glf\" (UniqueName: \"kubernetes.io/projected/a05c180a-04d3-4fa2-904d-034acaf5ada9-kube-api-access-r9glf\") pod \"node-resolver-97sf8\" (UID: \"a05c180a-04d3-4fa2-904d-034acaf5ada9\") " pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.733810 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.733599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnw9\" (UniqueName: \"kubernetes.io/projected/b585eadd-75f4-4d2c-828f-1a21849d59dd-kube-api-access-9gnw9\") pod \"aws-ebs-csi-driver-node-45drb\" (UID: \"b585eadd-75f4-4d2c-828f-1a21849d59dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.762283 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.762256    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:27:18.801710 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.801598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:18.811514 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.811488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkrjc"
Jul 17 21:27:18.814948 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.814902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dbcaea_ab7d_4abd_9894_4db5d4e15dba.slice/crio-1f686d06a2d8af30beed359374edd98fdd025ee090af295aabca2702098afd50 WatchSource:0}: Error finding container 1f686d06a2d8af30beed359374edd98fdd025ee090af295aabca2702098afd50: Status 404 returned error can't find the container with id 1f686d06a2d8af30beed359374edd98fdd025ee090af295aabca2702098afd50
Jul 17 21:27:18.822590 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.822567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l9c7k"
Jul 17 21:27:18.828201 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.827977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swkc4"
Jul 17 21:27:18.831538 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.830948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadad1485_282e_484c_ba4b_ff8bb8ffd410.slice/crio-65aacffa8755ca8029b48bc009e81c4a81cfa79314dc098e06903b644e645bb2 WatchSource:0}: Error finding container 65aacffa8755ca8029b48bc009e81c4a81cfa79314dc098e06903b644e645bb2: Status 404 returned error can't find the container with id 65aacffa8755ca8029b48bc009e81c4a81cfa79314dc098e06903b644e645bb2
Jul 17 21:27:18.835408 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.835388    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdc8bd8_1135_46ad_a9e9_4e06928fc07e.slice/crio-23f4971e54e02fbafe27e3ec05d820f4c9f06147ab80ab4444ad323193e4a9be WatchSource:0}: Error finding container 23f4971e54e02fbafe27e3ec05d820f4c9f06147ab80ab4444ad323193e4a9be: Status 404 returned error can't find the container with id 23f4971e54e02fbafe27e3ec05d820f4c9f06147ab80ab4444ad323193e4a9be
Jul 17 21:27:18.836442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.836422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zm8fg"
Jul 17 21:27:18.838166 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.838140    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e3694e_96d4_4374_b47e_b666fc83db50.slice/crio-bfa36b4b53c71082945f60d217e686e70ad6c7f8a77c4da8773018eb55e6962a WatchSource:0}: Error finding container bfa36b4b53c71082945f60d217e686e70ad6c7f8a77c4da8773018eb55e6962a: Status 404 returned error can't find the container with id bfa36b4b53c71082945f60d217e686e70ad6c7f8a77c4da8773018eb55e6962a
Jul 17 21:27:18.845755 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.845720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2887a37_5f1f_42bd_b9ad_36a419bf1cdd.slice/crio-9ecafc7d91a242dd6e91fc88c09cdff78fe4020abf3a7c3021595d80a410c956 WatchSource:0}: Error finding container 9ecafc7d91a242dd6e91fc88c09cdff78fe4020abf3a7c3021595d80a410c956: Status 404 returned error can't find the container with id 9ecafc7d91a242dd6e91fc88c09cdff78fe4020abf3a7c3021595d80a410c956
Jul 17 21:27:18.845871 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.845816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-27mht"
Jul 17 21:27:18.855050 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.854881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:18.856248 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.856224    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40814c12_245a_4369_bfc8_4ea9cd93e60f.slice/crio-a8e9f69555160d00ee525c165e2a69dfbdb2d913ce08333d5d5cdab6bd831e01 WatchSource:0}: Error finding container a8e9f69555160d00ee525c165e2a69dfbdb2d913ce08333d5d5cdab6bd831e01: Status 404 returned error can't find the container with id a8e9f69555160d00ee525c165e2a69dfbdb2d913ce08333d5d5cdab6bd831e01
Jul 17 21:27:18.864190 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.864168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb"
Jul 17 21:27:18.864306 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.864264    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b0098e_e168_452f_a057_d6488676bb28.slice/crio-8f8ccb1e63509034a2d3f709b6bfe1d8bdec89a7b7e0de7e3bd40cf342fc3f49 WatchSource:0}: Error finding container 8f8ccb1e63509034a2d3f709b6bfe1d8bdec89a7b7e0de7e3bd40cf342fc3f49: Status 404 returned error can't find the container with id 8f8ccb1e63509034a2d3f709b6bfe1d8bdec89a7b7e0de7e3bd40cf342fc3f49
Jul 17 21:27:18.870769 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:18.870733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97sf8"
Jul 17 21:27:18.877132 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:18.877106    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb585eadd_75f4_4d2c_828f_1a21849d59dd.slice/crio-1b2ee1688944897c52446d71b8a67b32fa6d21ef9d86c88e18a6f74191b616e1 WatchSource:0}: Error finding container 1b2ee1688944897c52446d71b8a67b32fa6d21ef9d86c88e18a6f74191b616e1: Status 404 returned error can't find the container with id 1b2ee1688944897c52446d71b8a67b32fa6d21ef9d86c88e18a6f74191b616e1
Jul 17 21:27:19.119444 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.119326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:19.119652 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.119485    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:19.119652 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.119558    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:20.11953798 +0000 UTC m=+4.031778255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:19.220761 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.220718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:19.220946 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.220825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:19.220946 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.220888    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:19.220946 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.220915    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:19.220946 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.220938    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:19.221202 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.220957    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:19.221202 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.221002    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:27:20.220986684 +0000 UTC m=+4.133226957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:19.221202 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.221023    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:27:20.221013652 +0000 UTC m=+4.133253921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:19.563598 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.563514    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:22:19 +0000 UTC" deadline="2028-03-09 04:58:06.764611636 +0000 UTC"
Jul 17 21:27:19.563598 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.563552    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14407h30m47.201065212s"
Jul 17 21:27:19.597447 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.596910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:19.597447 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:19.597045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:19.610503 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.610420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97sf8" event={"ID":"a05c180a-04d3-4fa2-904d-034acaf5ada9","Type":"ContainerStarted","Data":"d454042622da0fdfdee21e8f9f2b50ecf116746e12f46fb7ce093445a06e71c9"}
Jul 17 21:27:19.612948 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.612864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb" event={"ID":"b585eadd-75f4-4d2c-828f-1a21849d59dd","Type":"ContainerStarted","Data":"1b2ee1688944897c52446d71b8a67b32fa6d21ef9d86c88e18a6f74191b616e1"}
Jul 17 21:27:19.615334 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.615305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sqnqz" event={"ID":"76b0098e-e168-452f-a057-d6488676bb28","Type":"ContainerStarted","Data":"8f8ccb1e63509034a2d3f709b6bfe1d8bdec89a7b7e0de7e3bd40cf342fc3f49"}
Jul 17 21:27:19.616665 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.616640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zm8fg" event={"ID":"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd","Type":"ContainerStarted","Data":"9ecafc7d91a242dd6e91fc88c09cdff78fe4020abf3a7c3021595d80a410c956"}
Jul 17 21:27:19.618132 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.618088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerStarted","Data":"a8e9f69555160d00ee525c165e2a69dfbdb2d913ce08333d5d5cdab6bd831e01"}
Jul 17 21:27:19.619489 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.619461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swkc4" event={"ID":"90e3694e-96d4-4374-b47e-b666fc83db50","Type":"ContainerStarted","Data":"bfa36b4b53c71082945f60d217e686e70ad6c7f8a77c4da8773018eb55e6962a"}
Jul 17 21:27:19.622074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.622041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l9c7k" event={"ID":"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e","Type":"ContainerStarted","Data":"23f4971e54e02fbafe27e3ec05d820f4c9f06147ab80ab4444ad323193e4a9be"}
Jul 17 21:27:19.623655 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.623580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkrjc" event={"ID":"adad1485-282e-484c-ba4b-ff8bb8ffd410","Type":"ContainerStarted","Data":"65aacffa8755ca8029b48bc009e81c4a81cfa79314dc098e06903b644e645bb2"}
Jul 17 21:27:19.625737 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:19.625639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"1f686d06a2d8af30beed359374edd98fdd025ee090af295aabca2702098afd50"}
Jul 17 21:27:20.130384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.129720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:20.130384 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.129867    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:20.130384 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.129935    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:22.129915911 +0000 UTC m=+6.042156197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:20.230692 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.230378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:20.230692 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.230460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:20.230692 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.230552    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:20.230692 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.230600    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:20.230692 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.230617    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:20.230692 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.230632    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:20.231159 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.231042    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:27:22.230606339 +0000 UTC m=+6.142846611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:20.231844 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.231602    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:27:22.231576383 +0000 UTC m=+6.143816672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:20.566316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.566192    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:22:19 +0000 UTC" deadline="2028-04-27 05:16:12.962133276 +0000 UTC"
Jul 17 21:27:20.566316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.566260    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15583h48m52.395877441s"
Jul 17 21:27:20.604057 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.604025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:20.604234 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.604157    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:20.604576 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:20.604554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:20.604729 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:20.604708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:21.597028 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:21.596463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:21.597028 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:21.596607    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:22.146896 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:22.146851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:22.147085 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.147068    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:22.147144 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.147137    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:26.147116679 +0000 UTC m=+10.059356972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:22.248027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:22.248119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.248284    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.248306    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.248322    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.248383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:27:26.248363561 +0000 UTC m=+10.160603848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.248463    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:22.248870 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.248498    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:27:26.248485173 +0000 UTC m=+10.160725442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:22.605396 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:22.605317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:22.605872 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.605454    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:22.605872 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:22.605509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:22.605872 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:22.605661    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:23.597268 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:23.596757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:23.597268 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:23.596910    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:24.605367 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:24.605134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:24.606525 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:24.605996    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:24.606525 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:24.606385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:24.606525 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:24.606483    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:25.596855 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:25.596815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:25.597063 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:25.596951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:26.182022 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:26.181345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:26.182022 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.181510    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:26.182022 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.181619    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:34.181581535 +0000 UTC m=+18.093821810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:26.282568 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:26.282359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:26.282568 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:26.282459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:26.282568 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.282564    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:26.282907 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.282646    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:26.282907 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.282674    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:27:34.282651323 +0000 UTC m=+18.194891608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:26.282907 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.282675    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:26.282907 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.282697    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:26.282907 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.282752    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:27:34.282734967 +0000 UTC m=+18.194975258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:26.600081 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:26.599642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:26.600081 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.599759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:26.600081 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:26.599829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:26.600081 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:26.599936    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:27.597229 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:27.597194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:27.597868 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:27.597299    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:28.596733 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:28.596697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:28.596911 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:28.596700    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:28.596911 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:28.596851    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:28.597020 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:28.596917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:29.597204 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:29.597168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:29.597687 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:29.597307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:30.596994 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:30.596782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:30.597175 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:30.596805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:30.597175 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:30.597121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:30.597259 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:30.597179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:31.597161 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:31.597119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:31.597353 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:31.597236    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:32.597110 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:32.596815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:32.597110 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:32.596961    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:32.597110 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:32.597002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:32.597110 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:32.597078    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:33.596761 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:33.596728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:33.596985 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:33.596868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:34.247173 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:34.247128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:34.247742 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.247292    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:34.247742 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.247353    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:50.247337686 +0000 UTC m=+34.159577952 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:34.347479 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:34.347440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:34.347699 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:34.347519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:34.347699 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.347573    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:34.347699 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.347660    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:27:50.347643672 +0000 UTC m=+34.259883942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:34.347699 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.347669    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:34.347699 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.347685    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:34.347699 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.347696    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:34.347933 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.347753    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:27:50.34773596 +0000 UTC m=+34.259976245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:34.596552 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:34.596470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:34.596552 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:34.596485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:34.596776 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.596619    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:34.596776 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:34.596728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:35.596394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:35.596359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:35.596834 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:35.596480    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:36.597970 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:36.597938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:36.598531 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:36.598500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:36.598973 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:36.598950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:36.599076 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:36.599060    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:37.596308 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:37.596266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:37.596505 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:37.596399    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:38.596690 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:38.596657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:38.597096 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:38.596790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:38.597096 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:38.596828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:38.597096 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:38.596913    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:39.597318 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.597153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:39.597921 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:39.597385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:39.670576 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.670543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sqnqz" event={"ID":"76b0098e-e168-452f-a057-d6488676bb28","Type":"ContainerStarted","Data":"41d8a1c8ae519a24f89f592566018233e511fcf840ce804760f1043a7333ebd6"}
Jul 17 21:27:39.672039 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.672013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" event={"ID":"1a28b61f94e609f57bdc8bf0691328b0","Type":"ContainerStarted","Data":"d1527c6fefc3b4f699ff7dc095e1ed236ca89066f124e76001243b032ae8a507"}
Jul 17 21:27:39.673644 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.673619    2563 generic.go:356] "Generic (PLEG): container finished" podID="40814c12-245a-4369-bfc8-4ea9cd93e60f" containerID="7cc160fa71276d687a81fa6a4bd8fa140101d0b0e00e20faa772ca566d17726b" exitCode=0
Jul 17 21:27:39.673739 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.673696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerDied","Data":"7cc160fa71276d687a81fa6a4bd8fa140101d0b0e00e20faa772ca566d17726b"}
Jul 17 21:27:39.675250 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.675229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swkc4" event={"ID":"90e3694e-96d4-4374-b47e-b666fc83db50","Type":"ContainerStarted","Data":"70f641e3ea61a9d19e474bed568dc5e7a722513e7f18c6869467e776ebb92abc"}
Jul 17 21:27:39.676759 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.676740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l9c7k" event={"ID":"4fdc8bd8-1135-46ad-a9e9-4e06928fc07e","Type":"ContainerStarted","Data":"d600a3614de82edae6268d00fb27d481ce250a4dc03e4023c7a2d20f65b652e1"}
Jul 17 21:27:39.678142 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.678117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkrjc" event={"ID":"adad1485-282e-484c-ba4b-ff8bb8ffd410","Type":"ContainerStarted","Data":"4c48308e48771d5e65e0db4ea958d539ed638712cd5ac4b2699f8e9ee8247fca"}
Jul 17 21:27:39.680423 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.680399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"957ac90766e44162af8f8cedabb109eafdffc028e75f224a79db62c83419983c"}
Jul 17 21:27:39.680499 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.680473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"22a13d8e22160db9c16b6621d6ca8713f8790438ba312f47433a1b147a8e8f05"}
Jul 17 21:27:39.680548 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.680525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"9fdb7ec1b589d26865690d5d6e8683e0455921eed57af4604891e0cbdb964c5d"}
Jul 17 21:27:39.681882 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.681864    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab437d1b1f10a17975f1fc588a4bc307" containerID="50a562abed567facac55a6c7995606aeba996837757b905e0aeeb359a5002915" exitCode=0
Jul 17 21:27:39.681977 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.681932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"ab437d1b1f10a17975f1fc588a4bc307","Type":"ContainerDied","Data":"50a562abed567facac55a6c7995606aeba996837757b905e0aeeb359a5002915"}
Jul 17 21:27:39.683321 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.683298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97sf8" event={"ID":"a05c180a-04d3-4fa2-904d-034acaf5ada9","Type":"ContainerStarted","Data":"7143ea051750f4c7d164c11532e940146e162b5f0d406af4bb225772f37fb2cd"}
Jul 17 21:27:39.684622 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.684586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb" event={"ID":"b585eadd-75f4-4d2c-828f-1a21849d59dd","Type":"ContainerStarted","Data":"7a5b8e1851c4d9872f5ba7888486c7faea36a366fd0dd62f5f359f2514b526b7"}
Jul 17 21:27:39.692543 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.692492    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sqnqz" podStartSLOduration=7.189604622 podStartE2EDuration="22.692475744s" totalImagesPullingTime="15.502871122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:39.68894668 +0000 UTC m=+23.601186966" watchObservedRunningTime="2026-07-17 21:27:39.692475744 +0000 UTC m=+23.604716034"
Jul 17 21:27:39.709334 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.709300    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-97sf8" podStartSLOduration=2.4893530950000002 podStartE2EDuration="22.709287869s" totalImagesPullingTime="20.219934774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:39.709259872 +0000 UTC m=+23.621500160" watchObservedRunningTime="2026-07-17 21:27:39.709287869 +0000 UTC m=+23.621528159"
Jul 17 21:27:39.732529 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.732492    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" podStartSLOduration=21.732479736 podStartE2EDuration="21.732479736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:39.732146315 +0000 UTC m=+23.644386605" watchObservedRunningTime="2026-07-17 21:27:39.732479736 +0000 UTC m=+23.644720024"
Jul 17 21:27:39.796576 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.796504    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lkrjc" podStartSLOduration=2.520893266 podStartE2EDuration="22.796488122s" totalImagesPullingTime="20.275594856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:39.765326202 +0000 UTC m=+23.677566491" watchObservedRunningTime="2026-07-17 21:27:39.796488122 +0000 UTC m=+23.708728393"
Jul 17 21:27:39.796870 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.796845    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l9c7k" podStartSLOduration=2.530674732 podStartE2EDuration="22.796837194s" totalImagesPullingTime="20.266162462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:39.796584275 +0000 UTC m=+23.708824567" watchObservedRunningTime="2026-07-17 21:27:39.796837194 +0000 UTC m=+23.709077483"
Jul 17 21:27:39.921411 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:39.921364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-swkc4" podStartSLOduration=2.606905369 podStartE2EDuration="22.921347385s" totalImagesPullingTime="20.314442016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:39.873744089 +0000 UTC m=+23.785984378" watchObservedRunningTime="2026-07-17 21:27:39.921347385 +0000 UTC m=+23.833587674"
Jul 17 21:27:40.596708 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.596459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:40.596848 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.596522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:40.596848 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:40.596830    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:40.597004 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:40.596898    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:40.690300 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.690273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"cd2c1012729b709882f2f4133b2f87dbde6358c965677cc855d45c816e7bb3b2"}
Jul 17 21:27:40.690681 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.690359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"35bce432d7c86e236ab8b7bd2b330597810382595d6bede5e53bea71312d1e69"}
Jul 17 21:27:40.690681 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.690416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"6f317dc961fbeb84eb4ee86cc685a883af974bcfca420939e13317e9c5f5332c"}
Jul 17 21:27:40.691507 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.691485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zm8fg" event={"ID":"d2887a37-5f1f-42bd-b9ad-36a419bf1cdd","Type":"ContainerStarted","Data":"1cf480c1c1160dff0f8b72567430a19cc92b4abb39c102170b20fbc9bd877d68"}
Jul 17 21:27:40.769847 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:40.769820    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 21:27:41.578485 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.578369    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:27:40.769842988Z","UUID":"9e0d14c7-205d-4241-9b83-5a383213309b","Handler":null,"Name":"","Endpoint":""}
Jul 17 21:27:41.580223 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.580203    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 21:27:41.580341 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.580243    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 21:27:41.596408 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.596377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:41.596558 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:41.596475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:41.695612 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.695564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"ab437d1b1f10a17975f1fc588a4bc307","Type":"ContainerStarted","Data":"11f94f278e696c27fdc5b5ce90324b2356b97484683b275b93022c843e19de2c"}
Jul 17 21:27:41.697330 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.697307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb" event={"ID":"b585eadd-75f4-4d2c-828f-1a21849d59dd","Type":"ContainerStarted","Data":"7fbb846a0aff68e7c868ea378fcfddb5816ce0181b2f25e5e43d83c8b6c10b02"}
Jul 17 21:27:41.710791 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.710752    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" podStartSLOduration=23.710738931 podStartE2EDuration="23.710738931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:41.710712302 +0000 UTC m=+25.622952591" watchObservedRunningTime="2026-07-17 21:27:41.710738931 +0000 UTC m=+25.622979221"
Jul 17 21:27:41.711094 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:41.711075    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zm8fg" podStartSLOduration=4.453374895 podStartE2EDuration="24.711067263s" totalImagesPullingTime="20.257692368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:40.725931316 +0000 UTC m=+24.638171605" watchObservedRunningTime="2026-07-17 21:27:41.711067263 +0000 UTC m=+25.623307552"
Jul 17 21:27:42.199787 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.199744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:42.200478 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.200457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:42.596385 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.596302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:42.596385 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.596332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:42.596561 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:42.596445    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:42.596561 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:42.596464    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:42.702411 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.702371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"df7a5af943603c55d85f7d6504abc0a6257b6298c4f8cdda9aa2985a4589e919"}
Jul 17 21:27:42.704162 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.704138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb" event={"ID":"b585eadd-75f4-4d2c-828f-1a21849d59dd","Type":"ContainerStarted","Data":"a973d810d833d7dc645da0d418ca23643c67f6642d53ff421e5485c711901d59"}
Jul 17 21:27:42.704352 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.704336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:42.704810 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.704794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sqnqz"
Jul 17 21:27:42.720043 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:42.720006    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45drb" podStartSLOduration=11.991279238 podStartE2EDuration="25.719993592s" totalImagesPullingTime="13.728714354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:42.719781649 +0000 UTC m=+26.632021955" watchObservedRunningTime="2026-07-17 21:27:42.719993592 +0000 UTC m=+26.632233881"
Jul 17 21:27:43.596254 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:43.596220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:43.596442 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:43.596332    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:44.596983 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:44.596834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:44.597379 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:44.596878    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:44.597379 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:44.597098    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:44.597379 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:44.597167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:44.713114 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:44.712866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" event={"ID":"80dbcaea-ab7d-4abd-9894-4db5d4e15dba","Type":"ContainerStarted","Data":"7bfcf8fdb4e34a324c34e3857a0ebd934a0a9589fedb052caec49307a03e4fb0"}
Jul 17 21:27:45.596133 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.596105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:45.596307 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:45.596227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:45.716567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.716534    2563 generic.go:356] "Generic (PLEG): container finished" podID="40814c12-245a-4369-bfc8-4ea9cd93e60f" containerID="0bb723c7111979feb87a3bdc03d38f9321fc78e6cb558fc33d86ab55440c4512" exitCode=0
Jul 17 21:27:45.717495 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.716627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerDied","Data":"0bb723c7111979feb87a3bdc03d38f9321fc78e6cb558fc33d86ab55440c4512"}
Jul 17 21:27:45.717495 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.717059    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:45.717495 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.717122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:45.717495 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.717164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:45.734481 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.734447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:45.734631 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.734585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:27:45.740450 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:45.740383    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b" podStartSLOduration=8.378875171 podStartE2EDuration="28.74036335s" totalImagesPullingTime="20.361488179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:44.743125796 +0000 UTC m=+28.655366086" watchObservedRunningTime="2026-07-17 21:27:45.74036335 +0000 UTC m=+29.652603638"
Jul 17 21:27:46.597611 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.597381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:46.597611 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.597423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:46.597913 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:46.597720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:46.597913 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:46.597840    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:46.887801 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.887775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dz78x"]
Jul 17 21:27:46.888082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.887875    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:46.888082 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:46.887962    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:46.891173 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.891147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jqdbf"]
Jul 17 21:27:46.891307 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.891250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:46.891384 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:46.891352    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:46.892060 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.891802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4nctl"]
Jul 17 21:27:46.892060 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:46.891889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:46.892060 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:46.891968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:47.723589 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:47.723551    2563 generic.go:356] "Generic (PLEG): container finished" podID="40814c12-245a-4369-bfc8-4ea9cd93e60f" containerID="1ecc1154f8feea0651683d838c6e08b29cd968cac6b7394402f61151fa07230c" exitCode=0
Jul 17 21:27:47.723799 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:47.723617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerDied","Data":"1ecc1154f8feea0651683d838c6e08b29cd968cac6b7394402f61151fa07230c"}
Jul 17 21:27:48.597059 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:48.596993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:48.597384 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:48.597093    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:48.597384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:48.596993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:48.597384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:48.597145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:48.597384 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:48.597225    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:48.597384 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:48.597290    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:49.730667 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:49.730624    2563 generic.go:356] "Generic (PLEG): container finished" podID="40814c12-245a-4369-bfc8-4ea9cd93e60f" containerID="273c2df82b5bdfcd0a059b724a54bfc7a0ba135ed7f2c7cf2a66fb1858791782" exitCode=0
Jul 17 21:27:49.730667 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:49.730668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerDied","Data":"273c2df82b5bdfcd0a059b724a54bfc7a0ba135ed7f2c7cf2a66fb1858791782"}
Jul 17 21:27:50.265802 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:50.265768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:50.265998 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.265902    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:50.265998 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.265960    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret podName:0a956c0c-f929-4b1c-a9e1-7449f408c8a7 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:22.265942697 +0000 UTC m=+66.178182969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret") pod "global-pull-secret-syncer-dz78x" (UID: "0a956c0c-f929-4b1c-a9e1-7449f408c8a7") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:27:50.366850 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:50.366769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:50.366850 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:50.366820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:50.367023 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.366907    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:50.367023 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.366917    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:27:50.367023 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.366934    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:27:50.367023 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.366943    2563 projected.go:196] Error preparing data for projected volume kube-api-access-frkkw for pod openshift-network-diagnostics/network-check-target-4nctl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:50.367023 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.366963    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs podName:68e20af5-a9a5-464f-81d1-67a8c17db4cf nodeName:}" failed. No retries permitted until 2026-07-17 21:28:22.366947979 +0000 UTC m=+66.279188246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs") pod "network-metrics-daemon-jqdbf" (UID: "68e20af5-a9a5-464f-81d1-67a8c17db4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:27:50.367023 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.366978    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw podName:3841cca9-8ae0-4673-8009-13230454aa9b nodeName:}" failed. No retries permitted until 2026-07-17 21:28:22.366968144 +0000 UTC m=+66.279208414 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-frkkw" (UniqueName: "kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw") pod "network-check-target-4nctl" (UID: "3841cca9-8ae0-4673-8009-13230454aa9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:27:50.596550 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:50.596515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:50.596550 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:50.596530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:50.596550 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:50.596530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:50.596787 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.596665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqdbf" podUID="68e20af5-a9a5-464f-81d1-67a8c17db4cf"
Jul 17 21:27:50.596787 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.596763    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dz78x" podUID="0a956c0c-f929-4b1c-a9e1-7449f408c8a7"
Jul 17 21:27:50.596861 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:50.596834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nctl" podUID="3841cca9-8ae0-4673-8009-13230454aa9b"
Jul 17 21:27:51.888235 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.888064    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeReady"
Jul 17 21:27:51.888615 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.888321    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 21:27:51.925193 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.925159    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"]
Jul 17 21:27:51.926855 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.926836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"
Jul 17 21:27:51.929657 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.929457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nk752\""
Jul 17 21:27:51.931432 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.931413    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.931544 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.931459    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.932648 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.932632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-95s5k"]
Jul 17 21:27:51.936645 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.936627    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"]
Jul 17 21:27:51.937015 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.936995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:51.939311 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.938529    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7465848bb8-f277d"]
Jul 17 21:27:51.939311 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.938685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.940411    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.940715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.940764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.943233    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8rnft\""
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.943269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.943359    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.943780 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.943632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qc6rh\""
Jul 17 21:27:51.945346 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.944787    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"]
Jul 17 21:27:51.945346 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.944911    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 17 21:27:51.945346 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.945139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.945346 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.945268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 17 21:27:51.945346 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.945333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:51.948953 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.948930    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"]
Jul 17 21:27:51.949301 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.949284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:51.952478 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.952458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bd8xd\""
Jul 17 21:27:51.952844 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.952824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 21:27:51.953201 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.953180    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 21:27:51.953450 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.953435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 21:27:51.954579 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.954528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"]
Jul 17 21:27:51.954579 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.954563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"]
Jul 17 21:27:51.954755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.954584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-95s5k"]
Jul 17 21:27:51.954755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.954630    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b6cwm"]
Jul 17 21:27:51.955351 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.955331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.956423 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.956404    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 17 21:27:51.956683 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.956659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wt9pc\""
Jul 17 21:27:51.957014 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.956730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 17 21:27:51.957122 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.957069    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-psfq2"]
Jul 17 21:27:51.958321 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.957179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.958321 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.957622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:51.958321 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.958249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:51.960277 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.959656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:51.960277 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.959679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 17 21:27:51.960277 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.959955    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"]
Jul 17 21:27:51.962293 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.961664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7465848bb8-f277d"]
Jul 17 21:27:51.962293 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.961698    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"]
Jul 17 21:27:51.962293 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.961761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"
Jul 17 21:27:51.964203 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.964184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:51.967008 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.966986    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 21:27:51.967335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.967311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.967960 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.967939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.968061 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.968218 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 21:27:51.968303 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 17 21:27:51.968426 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968413    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 17 21:27:51.968483 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qt4k4\""
Jul 17 21:27:51.968680 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968665    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 21:27:51.968761 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2n6kt\""
Jul 17 21:27:51.968818 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 17 21:27:51.968928 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-brlsr\""
Jul 17 21:27:51.969044 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.968956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.969177 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.969159    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dhbkk\""
Jul 17 21:27:51.969253 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.969223    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:27:51.969310 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.969242    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 21:27:51.969364 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.969354    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:27:51.970033 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.970010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6b9d7\""
Jul 17 21:27:51.980679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.980660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-psfq2"]
Jul 17 21:27:51.984353 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.983889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b6cwm"]
Jul 17 21:27:51.985555 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.985535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"]
Jul 17 21:27:51.992126 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.991666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"]
Jul 17 21:27:51.993473 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.993456    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"]
Jul 17 21:27:51.994826 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:51.994803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"]
Jul 17 21:27:52.076509 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-trusted-ca\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d9fef3e-0416-41cd-8c83-831674459a8c-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee5cb13c-da63-472a-816d-71d66eeeeda9-config\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-certificates\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5cb13c-da63-472a-816d-71d66eeeeda9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-serving-cert\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrgp\" (UniqueName: \"kubernetes.io/projected/600ae2a5-5709-473f-b2c1-63145e434e91-kube-api-access-mhrgp\") pod \"ingress-canary-psfq2\" (UID: \"600ae2a5-5709-473f-b2c1-63145e434e91\") " pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.076715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76d7418b-0187-4848-a3e1-8fcbab28dc4b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.076996 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9d5559-38df-4b36-90d6-3c572fb461d4-tmp-dir\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.076996 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-bound-sa-token\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.076996 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tlk\" (UniqueName: \"kubernetes.io/projected/fbed4061-adc3-4a72-98e1-ed314e75909e-kube-api-access-w6tlk\") pod \"volume-data-source-validator-78cf4c6f6-4nwfb\" (UID: \"fbed4061-adc3-4a72-98e1-ed314e75909e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"
Jul 17 21:27:52.076996 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/600ae2a5-5709-473f-b2c1-63145e434e91-cert\") pod \"ingress-canary-psfq2\" (UID: \"600ae2a5-5709-473f-b2c1-63145e434e91\") " pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.076996 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr59f\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-kube-api-access-tr59f\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.076996 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.076957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-ca-trust-extracted\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-image-registry-private-configuration\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvs8\" (UniqueName: \"kubernetes.io/projected/27a8a11c-2171-4fa0-b009-b70206d4fabd-kube-api-access-9zvs8\") pod \"network-check-source-7bc5b6bf85-7l2l4\" (UID: \"27a8a11c-2171-4fa0-b009-b70206d4fabd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2sm\" (UniqueName: \"kubernetes.io/projected/76d7418b-0187-4848-a3e1-8fcbab28dc4b-kube-api-access-fb2sm\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76d7418b-0187-4848-a3e1-8fcbab28dc4b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-trusted-ca\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.077237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-installation-pull-secrets\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-config\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gwk\" (UniqueName: \"kubernetes.io/projected/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-kube-api-access-n7gwk\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brck\" (UniqueName: \"kubernetes.io/projected/534d8a6e-9bad-4091-b341-10bc20d9dd6e-kube-api-access-7brck\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9d5559-38df-4b36-90d6-3c572fb461d4-config-volume\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rnn\" (UniqueName: \"kubernetes.io/projected/ee5cb13c-da63-472a-816d-71d66eeeeda9-kube-api-access-m7rnn\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.077558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.077539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqrv\" (UniqueName: \"kubernetes.io/projected/5f9d5559-38df-4b36-90d6-3c572fb461d4-kube-api-access-dpqrv\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.128175 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.128134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqnqz_76b0098e-e168-452f-a057-d6488676bb28/konnectivity-agent/0.log" limit=1024
Jul 17 21:27:52.178456 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d9fef3e-0416-41cd-8c83-831674459a8c-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:52.178456 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee5cb13c-da63-472a-816d-71d66eeeeda9-config\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.178456 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-certificates\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.178456 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5cb13c-da63-472a-816d-71d66eeeeda9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-serving-cert\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrgp\" (UniqueName: \"kubernetes.io/projected/600ae2a5-5709-473f-b2c1-63145e434e91-kube-api-access-mhrgp\") pod \"ingress-canary-psfq2\" (UID: \"600ae2a5-5709-473f-b2c1-63145e434e91\") " pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76d7418b-0187-4848-a3e1-8fcbab28dc4b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9d5559-38df-4b36-90d6-3c572fb461d4-tmp-dir\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-bound-sa-token\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tlk\" (UniqueName: \"kubernetes.io/projected/fbed4061-adc3-4a72-98e1-ed314e75909e-kube-api-access-w6tlk\") pod \"volume-data-source-validator-78cf4c6f6-4nwfb\" (UID: \"fbed4061-adc3-4a72-98e1-ed314e75909e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/600ae2a5-5709-473f-b2c1-63145e434e91-cert\") pod \"ingress-canary-psfq2\" (UID: \"600ae2a5-5709-473f-b2c1-63145e434e91\") " pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr59f\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-kube-api-access-tr59f\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-ca-trust-extracted\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.178779 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-image-registry-private-configuration\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvs8\" (UniqueName: \"kubernetes.io/projected/27a8a11c-2171-4fa0-b009-b70206d4fabd-kube-api-access-9zvs8\") pod \"network-check-source-7bc5b6bf85-7l2l4\" (UID: \"27a8a11c-2171-4fa0-b009-b70206d4fabd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2sm\" (UniqueName: \"kubernetes.io/projected/76d7418b-0187-4848-a3e1-8fcbab28dc4b-kube-api-access-fb2sm\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76d7418b-0187-4848-a3e1-8fcbab28dc4b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-trusted-ca\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-installation-pull-secrets\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-config\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.178986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gwk\" (UniqueName: \"kubernetes.io/projected/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-kube-api-access-n7gwk\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brck\" (UniqueName: \"kubernetes.io/projected/534d8a6e-9bad-4091-b341-10bc20d9dd6e-kube-api-access-7brck\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9d5559-38df-4b36-90d6-3c572fb461d4-config-volume\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rnn\" (UniqueName: \"kubernetes.io/projected/ee5cb13c-da63-472a-816d-71d66eeeeda9-kube-api-access-m7rnn\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqrv\" (UniqueName: \"kubernetes.io/projected/5f9d5559-38df-4b36-90d6-3c572fb461d4-kube-api-access-dpqrv\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.179251 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-trusted-ca\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.179940 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee5cb13c-da63-472a-816d-71d66eeeeda9-config\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.179940 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d9fef3e-0416-41cd-8c83-831674459a8c-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:52.179940 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-certificates\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.179940 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.179550    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:27:52.179940 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.179629    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls podName:534d8a6e-9bad-4091-b341-10bc20d9dd6e nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.679589328 +0000 UTC m=+36.591829594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wvcgz" (UID: "534d8a6e-9bad-4091-b341-10bc20d9dd6e") : secret "samples-operator-tls" not found
Jul 17 21:27:52.180154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.179987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-trusted-ca\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.180154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.180085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-ca-trust-extracted\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.180255 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.180172    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:27:52.180255 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.180185    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7465848bb8-f277d: secret "image-registry-tls" not found
Jul 17 21:27:52.180255 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.180223    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls podName:522f30fd-2eb0-47e9-a6c1-0ee6e31a893a nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.680209855 +0000 UTC m=+36.592450136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls") pod "image-registry-7465848bb8-f277d" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a") : secret "image-registry-tls" not found
Jul 17 21:27:52.180947 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.180702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76d7418b-0187-4848-a3e1-8fcbab28dc4b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.180947 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.180907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9d5559-38df-4b36-90d6-3c572fb461d4-tmp-dir\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.181158 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.181028    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:27:52.181158 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.181083    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls podName:5f9d5559-38df-4b36-90d6-3c572fb461d4 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.681068449 +0000 UTC m=+36.593308728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls") pod "dns-default-b6cwm" (UID: "5f9d5559-38df-4b36-90d6-3c572fb461d4") : secret "dns-default-metrics-tls" not found
Jul 17 21:27:52.181158 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.181149    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:27:52.181379 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.181186    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert podName:8d9fef3e-0416-41cd-8c83-831674459a8c nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.681174712 +0000 UTC m=+36.593414992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5njn5" (UID: "8d9fef3e-0416-41cd-8c83-831674459a8c") : secret "networking-console-plugin-cert" not found
Jul 17 21:27:52.181379 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.181345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-config\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.181843 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.181815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-trusted-ca\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.181843 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.181841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9d5559-38df-4b36-90d6-3c572fb461d4-config-volume\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.184206 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.184154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76d7418b-0187-4848-a3e1-8fcbab28dc4b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.184206 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.184153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5cb13c-da63-472a-816d-71d66eeeeda9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.184369 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.184270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-installation-pull-secrets\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.184462 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.184441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-serving-cert\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.184571 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.184507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-image-registry-private-configuration\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.184691 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.184669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/600ae2a5-5709-473f-b2c1-63145e434e91-cert\") pod \"ingress-canary-psfq2\" (UID: \"600ae2a5-5709-473f-b2c1-63145e434e91\") " pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.193550 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.193515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rnn\" (UniqueName: \"kubernetes.io/projected/ee5cb13c-da63-472a-816d-71d66eeeeda9-kube-api-access-m7rnn\") pod \"service-ca-operator-5bcbc94c78-cmc9c\" (UID: \"ee5cb13c-da63-472a-816d-71d66eeeeda9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.196049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.195477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2sm\" (UniqueName: \"kubernetes.io/projected/76d7418b-0187-4848-a3e1-8fcbab28dc4b-kube-api-access-fb2sm\") pod \"kube-storage-version-migrator-operator-7658d9b749-lsnt6\" (UID: \"76d7418b-0187-4848-a3e1-8fcbab28dc4b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.196049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.195497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-bound-sa-token\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.196049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.195827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gwk\" (UniqueName: \"kubernetes.io/projected/a0dd0ed2-41e2-4790-8122-7ea4cc680c95-kube-api-access-n7gwk\") pod \"console-operator-565b7b569d-95s5k\" (UID: \"a0dd0ed2-41e2-4790-8122-7ea4cc680c95\") " pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.196049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.195905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvs8\" (UniqueName: \"kubernetes.io/projected/27a8a11c-2171-4fa0-b009-b70206d4fabd-kube-api-access-9zvs8\") pod \"network-check-source-7bc5b6bf85-7l2l4\" (UID: \"27a8a11c-2171-4fa0-b009-b70206d4fabd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"
Jul 17 21:27:52.196049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.196010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqrv\" (UniqueName: \"kubernetes.io/projected/5f9d5559-38df-4b36-90d6-3c572fb461d4-kube-api-access-dpqrv\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.196049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.196039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tlk\" (UniqueName: \"kubernetes.io/projected/fbed4061-adc3-4a72-98e1-ed314e75909e-kube-api-access-w6tlk\") pod \"volume-data-source-validator-78cf4c6f6-4nwfb\" (UID: \"fbed4061-adc3-4a72-98e1-ed314e75909e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"
Jul 17 21:27:52.196404 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.196074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr59f\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-kube-api-access-tr59f\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.196563 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.196541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brck\" (UniqueName: \"kubernetes.io/projected/534d8a6e-9bad-4091-b341-10bc20d9dd6e-kube-api-access-7brck\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:52.197269 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.197243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrgp\" (UniqueName: \"kubernetes.io/projected/600ae2a5-5709-473f-b2c1-63145e434e91-kube-api-access-mhrgp\") pod \"ingress-canary-psfq2\" (UID: \"600ae2a5-5709-473f-b2c1-63145e434e91\") " pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.210922 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.209993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqnqz_76b0098e-e168-452f-a057-d6488676bb28/konnectivity-agent/0.log" limit=1024
Jul 17 21:27:52.239897 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.239869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"
Jul 17 21:27:52.254939 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.254916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:27:52.266796 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.266769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"
Jul 17 21:27:52.283402 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.283222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"
Jul 17 21:27:52.314819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.311367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-psfq2"
Jul 17 21:27:52.323625 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.321246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"
Jul 17 21:27:52.465820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.465541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb"]
Jul 17 21:27:52.472489 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:52.472445    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbed4061_adc3_4a72_98e1_ed314e75909e.slice/crio-298b38012a5a3f8b5ce9464f9e812831f120a741f351966da0692038cf4b6f44 WatchSource:0}: Error finding container 298b38012a5a3f8b5ce9464f9e812831f120a741f351966da0692038cf4b6f44: Status 404 returned error can't find the container with id 298b38012a5a3f8b5ce9464f9e812831f120a741f351966da0692038cf4b6f44
Jul 17 21:27:52.482747 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.481695    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-95s5k"]
Jul 17 21:27:52.492864 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.492824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c"]
Jul 17 21:27:52.496233 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:52.496208    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5cb13c_da63_472a_816d_71d66eeeeda9.slice/crio-9de59ea7cceae5af65b73c1607aa9a297d3b507746471b5652cd486313606478 WatchSource:0}: Error finding container 9de59ea7cceae5af65b73c1607aa9a297d3b507746471b5652cd486313606478: Status 404 returned error can't find the container with id 9de59ea7cceae5af65b73c1607aa9a297d3b507746471b5652cd486313606478
Jul 17 21:27:52.513927 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.513833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6"]
Jul 17 21:27:52.517146 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:52.517118    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d7418b_0187_4848_a3e1_8fcbab28dc4b.slice/crio-0d48d668604d7671580772243712519e266acd193640f54d04d7f08a27c0775e WatchSource:0}: Error finding container 0d48d668604d7671580772243712519e266acd193640f54d04d7f08a27c0775e: Status 404 returned error can't find the container with id 0d48d668604d7671580772243712519e266acd193640f54d04d7f08a27c0775e
Jul 17 21:27:52.556717 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.556693    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-psfq2"]
Jul 17 21:27:52.559096 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:52.559067    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600ae2a5_5709_473f_b2c1_63145e434e91.slice/crio-3421f2e2ae3ea4aa86e81b72237af4d6cab898f9f4e920f81e061f2fbe8a2358 WatchSource:0}: Error finding container 3421f2e2ae3ea4aa86e81b72237af4d6cab898f9f4e920f81e061f2fbe8a2358: Status 404 returned error can't find the container with id 3421f2e2ae3ea4aa86e81b72237af4d6cab898f9f4e920f81e061f2fbe8a2358
Jul 17 21:27:52.562418 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.562394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4"]
Jul 17 21:27:52.565664 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:27:52.565641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a8a11c_2171_4fa0_b009_b70206d4fabd.slice/crio-b9f0727b1981ec20d8f9c8918e0e2a0a0deb25061321f69d646878bd5bbd666a WatchSource:0}: Error finding container b9f0727b1981ec20d8f9c8918e0e2a0a0deb25061321f69d646878bd5bbd666a: Status 404 returned error can't find the container with id b9f0727b1981ec20d8f9c8918e0e2a0a0deb25061321f69d646878bd5bbd666a
Jul 17 21:27:52.597523 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.597493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:27:52.601978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.597815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:27:52.601978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.598043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:27:52.601978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.600385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:27:52.601978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.600834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fkx7\""
Jul 17 21:27:52.601978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.600855    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7k8fl\""
Jul 17 21:27:52.601978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.601208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:27:52.687837 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.687638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:52.688006 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.687860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:52.688006 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.687793    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:27:52.688006 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.687940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:52.688006 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.687974    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert podName:8d9fef3e-0416-41cd-8c83-831674459a8c nodeName:}" failed. No retries permitted until 2026-07-17 21:27:53.687954018 +0000 UTC m=+37.600194286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5njn5" (UID: "8d9fef3e-0416-41cd-8c83-831674459a8c") : secret "networking-console-plugin-cert" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.688019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688037    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688048    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7465848bb8-f277d: secret "image-registry-tls" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688068    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls podName:522f30fd-2eb0-47e9-a6c1-0ee6e31a893a nodeName:}" failed. No retries permitted until 2026-07-17 21:27:53.688077118 +0000 UTC m=+37.600317401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls") pod "image-registry-7465848bb8-f277d" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a") : secret "image-registry-tls" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688122    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688174    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls podName:534d8a6e-9bad-4091-b341-10bc20d9dd6e nodeName:}" failed. No retries permitted until 2026-07-17 21:27:53.688150286 +0000 UTC m=+37.600390574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wvcgz" (UID: "534d8a6e-9bad-4091-b341-10bc20d9dd6e") : secret "samples-operator-tls" not found
Jul 17 21:27:52.688237 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:52.688207    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls podName:5f9d5559-38df-4b36-90d6-3c572fb461d4 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:53.688191829 +0000 UTC m=+37.600432111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls") pod "dns-default-b6cwm" (UID: "5f9d5559-38df-4b36-90d6-3c572fb461d4") : secret "dns-default-metrics-tls" not found
Jul 17 21:27:52.740199 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.740072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4" event={"ID":"27a8a11c-2171-4fa0-b009-b70206d4fabd","Type":"ContainerStarted","Data":"b9f0727b1981ec20d8f9c8918e0e2a0a0deb25061321f69d646878bd5bbd666a"}
Jul 17 21:27:52.741410 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.741370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6" event={"ID":"76d7418b-0187-4848-a3e1-8fcbab28dc4b","Type":"ContainerStarted","Data":"0d48d668604d7671580772243712519e266acd193640f54d04d7f08a27c0775e"}
Jul 17 21:27:52.742668 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.742641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" event={"ID":"a0dd0ed2-41e2-4790-8122-7ea4cc680c95","Type":"ContainerStarted","Data":"2cabf9e70c799d14f10ab6d73658a3872682fefdf2ba7704e38992fc727c8fcc"}
Jul 17 21:27:52.743918 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.743895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb" event={"ID":"fbed4061-adc3-4a72-98e1-ed314e75909e","Type":"ContainerStarted","Data":"298b38012a5a3f8b5ce9464f9e812831f120a741f351966da0692038cf4b6f44"}
Jul 17 21:27:52.745114 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.745070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-psfq2" event={"ID":"600ae2a5-5709-473f-b2c1-63145e434e91","Type":"ContainerStarted","Data":"3421f2e2ae3ea4aa86e81b72237af4d6cab898f9f4e920f81e061f2fbe8a2358"}
Jul 17 21:27:52.746289 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:52.746271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c" event={"ID":"ee5cb13c-da63-472a-816d-71d66eeeeda9","Type":"ContainerStarted","Data":"9de59ea7cceae5af65b73c1607aa9a297d3b507746471b5652cd486313606478"}
Jul 17 21:27:53.699041 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:53.699007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:53.699056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:53.699146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:53.699185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.699351    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.699414    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls podName:534d8a6e-9bad-4091-b341-10bc20d9dd6e nodeName:}" failed. No retries permitted until 2026-07-17 21:27:55.699396973 +0000 UTC m=+39.611637255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wvcgz" (UID: "534d8a6e-9bad-4091-b341-10bc20d9dd6e") : secret "samples-operator-tls" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.699842    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.699893    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert podName:8d9fef3e-0416-41cd-8c83-831674459a8c nodeName:}" failed. No retries permitted until 2026-07-17 21:27:55.699876859 +0000 UTC m=+39.612117140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5njn5" (UID: "8d9fef3e-0416-41cd-8c83-831674459a8c") : secret "networking-console-plugin-cert" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.699950    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.699983    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls podName:5f9d5559-38df-4b36-90d6-3c572fb461d4 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:55.699971353 +0000 UTC m=+39.612211635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls") pod "dns-default-b6cwm" (UID: "5f9d5559-38df-4b36-90d6-3c572fb461d4") : secret "dns-default-metrics-tls" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.700051    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.700061    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7465848bb8-f277d: secret "image-registry-tls" not found
Jul 17 21:27:53.700112 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:53.700094    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls podName:522f30fd-2eb0-47e9-a6c1-0ee6e31a893a nodeName:}" failed. No retries permitted until 2026-07-17 21:27:55.700082948 +0000 UTC m=+39.612323228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls") pod "image-registry-7465848bb8-f277d" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a") : secret "image-registry-tls" not found
Jul 17 21:27:55.721264 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:55.721224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:55.721290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:55.721360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721378    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:55.721392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721403    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7465848bb8-f277d: secret "image-registry-tls" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721484    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721455    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721540    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls podName:522f30fd-2eb0-47e9-a6c1-0ee6e31a893a nodeName:}" failed. No retries permitted until 2026-07-17 21:27:59.721519729 +0000 UTC m=+43.633760016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls") pod "image-registry-7465848bb8-f277d" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a") : secret "image-registry-tls" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721458    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721560    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls podName:5f9d5559-38df-4b36-90d6-3c572fb461d4 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:59.721550472 +0000 UTC m=+43.633790741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls") pod "dns-default-b6cwm" (UID: "5f9d5559-38df-4b36-90d6-3c572fb461d4") : secret "dns-default-metrics-tls" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721624    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert podName:8d9fef3e-0416-41cd-8c83-831674459a8c nodeName:}" failed. No retries permitted until 2026-07-17 21:27:59.721584997 +0000 UTC m=+43.633825269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5njn5" (UID: "8d9fef3e-0416-41cd-8c83-831674459a8c") : secret "networking-console-plugin-cert" not found
Jul 17 21:27:55.721737 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:55.721647    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls podName:534d8a6e-9bad-4091-b341-10bc20d9dd6e nodeName:}" failed. No retries permitted until 2026-07-17 21:27:59.721637103 +0000 UTC m=+43.633877376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wvcgz" (UID: "534d8a6e-9bad-4091-b341-10bc20d9dd6e") : secret "samples-operator-tls" not found
Jul 17 21:27:56.874832 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:56.874797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqnqz_76b0098e-e168-452f-a057-d6488676bb28/konnectivity-agent/0.log" limit=1024
Jul 17 21:27:59.758531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:59.758489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:59.758552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:59.758634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758640    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758661    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7465848bb8-f277d: secret "image-registry-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:27:59.758668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls podName:522f30fd-2eb0-47e9-a6c1-0ee6e31a893a nodeName:}" failed. No retries permitted until 2026-07-17 21:28:07.758693624 +0000 UTC m=+51.670933916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls") pod "image-registry-7465848bb8-f277d" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a") : secret "image-registry-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758727    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758778    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758794    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls podName:534d8a6e-9bad-4091-b341-10bc20d9dd6e nodeName:}" failed. No retries permitted until 2026-07-17 21:28:07.758775902 +0000 UTC m=+51.671016186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wvcgz" (UID: "534d8a6e-9bad-4091-b341-10bc20d9dd6e") : secret "samples-operator-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758802    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758820    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls podName:5f9d5559-38df-4b36-90d6-3c572fb461d4 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:07.758805872 +0000 UTC m=+51.671046158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls") pod "dns-default-b6cwm" (UID: "5f9d5559-38df-4b36-90d6-3c572fb461d4") : secret "dns-default-metrics-tls" not found
Jul 17 21:27:59.759121 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:27:59.758856    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert podName:8d9fef3e-0416-41cd-8c83-831674459a8c nodeName:}" failed. No retries permitted until 2026-07-17 21:28:07.758840678 +0000 UTC m=+51.671080945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5njn5" (UID: "8d9fef3e-0416-41cd-8c83-831674459a8c") : secret "networking-console-plugin-cert" not found
Jul 17 21:28:01.775442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.775406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6" event={"ID":"76d7418b-0187-4848-a3e1-8fcbab28dc4b","Type":"ContainerStarted","Data":"2ef7fce8c87b51f4dc4d0d421e7d51f6cef4f2cdc0deae94751143d4ecea4102"}
Jul 17 21:28:01.777442 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.777421    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/0.log"
Jul 17 21:28:01.777568 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.777459    2563 generic.go:356] "Generic (PLEG): container finished" podID="a0dd0ed2-41e2-4790-8122-7ea4cc680c95" containerID="41c16207b063f8c88627f55e9fb9a60cea44cf451f1a586d0904b756da894f7a" exitCode=255
Jul 17 21:28:01.777568 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.777534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" event={"ID":"a0dd0ed2-41e2-4790-8122-7ea4cc680c95","Type":"ContainerDied","Data":"41c16207b063f8c88627f55e9fb9a60cea44cf451f1a586d0904b756da894f7a"}
Jul 17 21:28:01.777749 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.777727    2563 scope.go:117] "RemoveContainer" containerID="41c16207b063f8c88627f55e9fb9a60cea44cf451f1a586d0904b756da894f7a"
Jul 17 21:28:01.779457 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.779411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb" event={"ID":"fbed4061-adc3-4a72-98e1-ed314e75909e","Type":"ContainerStarted","Data":"bc772ecbe2a1b3169f4e4daf4cefc67b637cb2ebc0f9b69c5ca53e2190ce6626"}
Jul 17 21:28:01.781514 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.781471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-psfq2" event={"ID":"600ae2a5-5709-473f-b2c1-63145e434e91","Type":"ContainerStarted","Data":"ebfbad9ddb5490b05966478dab85f9cab0e215b6a3bc385be76e8b854cbd18eb"}
Jul 17 21:28:01.783007 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.782953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c" event={"ID":"ee5cb13c-da63-472a-816d-71d66eeeeda9","Type":"ContainerStarted","Data":"17a3ebfa932f9fc67f243311e7457d031421e2c7ad928a6757f5315d843d2415"}
Jul 17 21:28:01.785480 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.784749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4" event={"ID":"27a8a11c-2171-4fa0-b009-b70206d4fabd","Type":"ContainerStarted","Data":"67880b78bf6ed6a2b5891e9f38214a3bcb300491ba7ce38529235a1f1f373b36"}
Jul 17 21:28:01.787615 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.787481    2563 generic.go:356] "Generic (PLEG): container finished" podID="40814c12-245a-4369-bfc8-4ea9cd93e60f" containerID="286cc0fc97e93cb13da9f7a7383912607247e03a7282f853d850e80ea1938731" exitCode=0
Jul 17 21:28:01.787615 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.787511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerDied","Data":"286cc0fc97e93cb13da9f7a7383912607247e03a7282f853d850e80ea1938731"}
Jul 17 21:28:01.792399 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.792354    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6" podStartSLOduration=30.194606963 podStartE2EDuration="38.792337395s" totalImagesPullingTime="8.597730432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.791390945 +0000 UTC m=+45.703631236" watchObservedRunningTime="2026-07-17 21:28:01.792337395 +0000 UTC m=+45.704577686"
Jul 17 21:28:01.825509 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.825465    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-psfq2" podStartSLOduration=2.268766655 podStartE2EDuration="10.825446549s" totalImagesPullingTime="8.556679894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.824731049 +0000 UTC m=+45.736971385" watchObservedRunningTime="2026-07-17 21:28:01.825446549 +0000 UTC m=+45.737686837"
Jul 17 21:28:01.840408 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.840357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nwfb" podStartSLOduration=34.201500498 podStartE2EDuration="42.840335466s" totalImagesPullingTime="8.638834968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.839686979 +0000 UTC m=+45.751927288" watchObservedRunningTime="2026-07-17 21:28:01.840335466 +0000 UTC m=+45.752575755"
Jul 17 21:28:01.879765 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:01.879714    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c" podStartSLOduration=30.263565962 podStartE2EDuration="38.879695876s" totalImagesPullingTime="8.616129914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.879677535 +0000 UTC m=+45.791917825" watchObservedRunningTime="2026-07-17 21:28:01.879695876 +0000 UTC m=+45.791936166"
Jul 17 21:28:02.255760 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.255716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:28:02.255901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.255787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:28:02.795629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.795579    2563 generic.go:356] "Generic (PLEG): container finished" podID="40814c12-245a-4369-bfc8-4ea9cd93e60f" containerID="1718162ab688f5b8b964f4bee5e2922336fc8d64f050f941cc634b95ecdc99d0" exitCode=0
Jul 17 21:28:02.796057 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.795687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerDied","Data":"1718162ab688f5b8b964f4bee5e2922336fc8d64f050f941cc634b95ecdc99d0"}
Jul 17 21:28:02.797760 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.797738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:28:02.798198 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.798180    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/0.log"
Jul 17 21:28:02.798301 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.798218    2563 generic.go:356] "Generic (PLEG): container finished" podID="a0dd0ed2-41e2-4790-8122-7ea4cc680c95" containerID="582e4b66cdcc30dfab50cb1026ce7a3dc5fafe454fcbd03aafa373f404029104" exitCode=255
Jul 17 21:28:02.798301 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.798248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" event={"ID":"a0dd0ed2-41e2-4790-8122-7ea4cc680c95","Type":"ContainerDied","Data":"582e4b66cdcc30dfab50cb1026ce7a3dc5fafe454fcbd03aafa373f404029104"}
Jul 17 21:28:02.798412 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.798302    2563 scope.go:117] "RemoveContainer" containerID="41c16207b063f8c88627f55e9fb9a60cea44cf451f1a586d0904b756da894f7a"
Jul 17 21:28:02.798669 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.798499    2563 scope.go:117] "RemoveContainer" containerID="582e4b66cdcc30dfab50cb1026ce7a3dc5fafe454fcbd03aafa373f404029104"
Jul 17 21:28:02.798772 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:02.798682    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-95s5k_openshift-console-operator(a0dd0ed2-41e2-4790-8122-7ea4cc680c95)\"" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" podUID="a0dd0ed2-41e2-4790-8122-7ea4cc680c95"
Jul 17 21:28:02.824384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:02.824337    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7l2l4" podStartSLOduration=29.26271067 podStartE2EDuration="37.824321558s" totalImagesPullingTime="8.561610888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.902036579 +0000 UTC m=+45.814276867" watchObservedRunningTime="2026-07-17 21:28:02.824321558 +0000 UTC m=+46.736561869"
Jul 17 21:28:03.018273 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.018232    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"]
Jul 17 21:28:03.042876 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.042842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"]
Jul 17 21:28:03.043046 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.042970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"
Jul 17 21:28:03.045557 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.045534    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 21:28:03.045731 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.045692    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dzmfg\""
Jul 17 21:28:03.045913 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.045858    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 21:28:03.090063 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.090031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwf4l\" (UniqueName: \"kubernetes.io/projected/ac42b126-9d2b-47fa-8926-fa792552e396-kube-api-access-jwf4l\") pod \"migrator-7cc9c7ffd7-7s68h\" (UID: \"ac42b126-9d2b-47fa-8926-fa792552e396\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"
Jul 17 21:28:03.191001 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.190958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwf4l\" (UniqueName: \"kubernetes.io/projected/ac42b126-9d2b-47fa-8926-fa792552e396-kube-api-access-jwf4l\") pod \"migrator-7cc9c7ffd7-7s68h\" (UID: \"ac42b126-9d2b-47fa-8926-fa792552e396\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"
Jul 17 21:28:03.200553 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.200515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwf4l\" (UniqueName: \"kubernetes.io/projected/ac42b126-9d2b-47fa-8926-fa792552e396-kube-api-access-jwf4l\") pod \"migrator-7cc9c7ffd7-7s68h\" (UID: \"ac42b126-9d2b-47fa-8926-fa792552e396\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"
Jul 17 21:28:03.354496 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.354408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"
Jul 17 21:28:03.500869 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.500840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h"]
Jul 17 21:28:03.503746 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:03.503717    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac42b126_9d2b_47fa_8926_fa792552e396.slice/crio-fca91ddcebb98261cc88791b0fd70032b40f719f13e44ab90f28a974db23c354 WatchSource:0}: Error finding container fca91ddcebb98261cc88791b0fd70032b40f719f13e44ab90f28a974db23c354: Status 404 returned error can't find the container with id fca91ddcebb98261cc88791b0fd70032b40f719f13e44ab90f28a974db23c354
Jul 17 21:28:03.803132 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.803091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h" event={"ID":"ac42b126-9d2b-47fa-8926-fa792552e396","Type":"ContainerStarted","Data":"fca91ddcebb98261cc88791b0fd70032b40f719f13e44ab90f28a974db23c354"}
Jul 17 21:28:03.806289 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.806265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27mht" event={"ID":"40814c12-245a-4369-bfc8-4ea9cd93e60f","Type":"ContainerStarted","Data":"fc6ee34a9b14ab01b9a93ffde26a7826e2189c9ae0e6c6ff4f91adfb08dbb315"}
Jul 17 21:28:03.807683 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.807667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:28:03.807956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.807939    2563 scope.go:117] "RemoveContainer" containerID="582e4b66cdcc30dfab50cb1026ce7a3dc5fafe454fcbd03aafa373f404029104"
Jul 17 21:28:03.808120 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:03.808100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-95s5k_openshift-console-operator(a0dd0ed2-41e2-4790-8122-7ea4cc680c95)\"" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" podUID="a0dd0ed2-41e2-4790-8122-7ea4cc680c95"
Jul 17 21:28:03.828662 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:03.828620    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-27mht" podStartSLOduration=8.006704644 podStartE2EDuration="46.828586217s" totalImagesPullingTime="38.821881573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:03.826743298 +0000 UTC m=+47.738983722" watchObservedRunningTime="2026-07-17 21:28:03.828586217 +0000 UTC m=+47.740826514"
Jul 17 21:28:05.683650 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.683618    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-5kh4r"]
Jul 17 21:28:05.696341 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.696314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.697291 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.697269    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-5kh4r"]
Jul 17 21:28:05.699652 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.699632    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 21:28:05.700480 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.700457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 21:28:05.700572 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.700482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 21:28:05.700572 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.700483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 21:28:05.700572 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.700462    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-d4rqx\""
Jul 17 21:28:05.812483 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.812445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f515594f-7659-4f38-b435-fd71e2ad5ddd-signing-cabundle\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.812483 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.812484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd25x\" (UniqueName: \"kubernetes.io/projected/f515594f-7659-4f38-b435-fd71e2ad5ddd-kube-api-access-nd25x\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.812717 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.812669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f515594f-7659-4f38-b435-fd71e2ad5ddd-signing-key\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.817085 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.817055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h" event={"ID":"ac42b126-9d2b-47fa-8926-fa792552e396","Type":"ContainerStarted","Data":"ed0bbe910a5526e409fae1108a4e9fea62a547ece87828189651c24a99b6921a"}
Jul 17 21:28:05.817193 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.817107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h" event={"ID":"ac42b126-9d2b-47fa-8926-fa792552e396","Type":"ContainerStarted","Data":"a67536bd0adfd8bf43d00ae434b1ac72685731988165e062ee014dd6880e558a"}
Jul 17 21:28:05.833222 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.833178    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-7s68h" podStartSLOduration=0.996155519 podStartE2EDuration="2.833165038s" totalImagesPullingTime="1.837009519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:05.832277957 +0000 UTC m=+49.744518247" watchObservedRunningTime="2026-07-17 21:28:05.833165038 +0000 UTC m=+49.745405379"
Jul 17 21:28:05.914084 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.914046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f515594f-7659-4f38-b435-fd71e2ad5ddd-signing-key\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.914278 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.914108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f515594f-7659-4f38-b435-fd71e2ad5ddd-signing-cabundle\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.914278 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.914134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd25x\" (UniqueName: \"kubernetes.io/projected/f515594f-7659-4f38-b435-fd71e2ad5ddd-kube-api-access-nd25x\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.914916 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.914891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f515594f-7659-4f38-b435-fd71e2ad5ddd-signing-cabundle\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.916729 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.916707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f515594f-7659-4f38-b435-fd71e2ad5ddd-signing-key\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:05.922136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:05.922114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd25x\" (UniqueName: \"kubernetes.io/projected/f515594f-7659-4f38-b435-fd71e2ad5ddd-kube-api-access-nd25x\") pod \"service-ca-58598695f7-5kh4r\" (UID: \"f515594f-7659-4f38-b435-fd71e2ad5ddd\") " pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:06.005427 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:06.005352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-5kh4r"
Jul 17 21:28:06.142264 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:06.142230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-5kh4r"]
Jul 17 21:28:06.145806 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:06.145775    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf515594f_7659_4f38_b435_fd71e2ad5ddd.slice/crio-3095dd5dadd32b705de47774c63aa20d635db4ac2af0c8a7139275df933c3828 WatchSource:0}: Error finding container 3095dd5dadd32b705de47774c63aa20d635db4ac2af0c8a7139275df933c3828: Status 404 returned error can't find the container with id 3095dd5dadd32b705de47774c63aa20d635db4ac2af0c8a7139275df933c3828
Jul 17 21:28:06.821385 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:06.821350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-5kh4r" event={"ID":"f515594f-7659-4f38-b435-fd71e2ad5ddd","Type":"ContainerStarted","Data":"b986f7a7ade614b53d731225e69326f7d134a1a448f302a10fd5efaa755ed5ac"}
Jul 17 21:28:06.821772 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:06.821400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-5kh4r" event={"ID":"f515594f-7659-4f38-b435-fd71e2ad5ddd","Type":"ContainerStarted","Data":"3095dd5dadd32b705de47774c63aa20d635db4ac2af0c8a7139275df933c3828"}
Jul 17 21:28:06.839855 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:06.839809    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-5kh4r" podStartSLOduration=1.839794532 podStartE2EDuration="1.839794532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:06.838411916 +0000 UTC m=+50.750652207" watchObservedRunningTime="2026-07-17 21:28:06.839794532 +0000 UTC m=+50.752034820"
Jul 17 21:28:07.827987 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:07.827952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:28:07.827987 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:07.827999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:07.828079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:07.828117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828156    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828229    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls podName:5f9d5559-38df-4b36-90d6-3c572fb461d4 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:23.828209323 +0000 UTC m=+67.740449602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls") pod "dns-default-b6cwm" (UID: "5f9d5559-38df-4b36-90d6-3c572fb461d4") : secret "dns-default-metrics-tls" not found
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828245    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828292    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls podName:534d8a6e-9bad-4091-b341-10bc20d9dd6e nodeName:}" failed. No retries permitted until 2026-07-17 21:28:23.828277706 +0000 UTC m=+67.740517981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wvcgz" (UID: "534d8a6e-9bad-4091-b341-10bc20d9dd6e") : secret "samples-operator-tls" not found
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828352    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828362    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7465848bb8-f277d: secret "image-registry-tls" not found
Jul 17 21:28:07.828518 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828392    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls podName:522f30fd-2eb0-47e9-a6c1-0ee6e31a893a nodeName:}" failed. No retries permitted until 2026-07-17 21:28:23.828381738 +0000 UTC m=+67.740622008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls") pod "image-registry-7465848bb8-f277d" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a") : secret "image-registry-tls" not found
Jul 17 21:28:07.828971 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828613    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:28:07.828971 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:07.828660    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert podName:8d9fef3e-0416-41cd-8c83-831674459a8c nodeName:}" failed. No retries permitted until 2026-07-17 21:28:23.828645902 +0000 UTC m=+67.740886183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5njn5" (UID: "8d9fef3e-0416-41cd-8c83-831674459a8c") : secret "networking-console-plugin-cert" not found
Jul 17 21:28:12.255673 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.255629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:28:12.256258 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.255701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:28:12.256258 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.256022    2563 scope.go:117] "RemoveContainer" containerID="582e4b66cdcc30dfab50cb1026ce7a3dc5fafe454fcbd03aafa373f404029104"
Jul 17 21:28:12.845139 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.845108    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:28:12.845317 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.845170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" event={"ID":"a0dd0ed2-41e2-4790-8122-7ea4cc680c95","Type":"ContainerStarted","Data":"effadca7693aee6c78a861901dcbb721ad498d747e0f973352255650e2257c6e"}
Jul 17 21:28:12.845520 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.845493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:28:12.850581 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.850560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-95s5k"
Jul 17 21:28:12.888823 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:12.888770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-95s5k" podStartSLOduration=44.259472942 podStartE2EDuration="52.888751685s" totalImagesPullingTime="8.629278743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:12.88863457 +0000 UTC m=+56.800874884" watchObservedRunningTime="2026-07-17 21:28:12.888751685 +0000 UTC m=+56.800991972"
Jul 17 21:28:17.741669 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:17.741639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dh88b"
Jul 17 21:28:22.356846 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.356806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:28:22.359482 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.359465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:28:22.369688 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.369664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a956c0c-f929-4b1c-a9e1-7449f408c8a7-original-pull-secret\") pod \"global-pull-secret-syncer-dz78x\" (UID: \"0a956c0c-f929-4b1c-a9e1-7449f408c8a7\") " pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:28:22.457460 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.457427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:28:22.457660 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.457476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:28:22.460022 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.459999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkkw\" (UniqueName: \"kubernetes.io/projected/3841cca9-8ae0-4673-8009-13230454aa9b-kube-api-access-frkkw\") pod \"network-check-target-4nctl\" (UID: \"3841cca9-8ae0-4673-8009-13230454aa9b\") " pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:28:22.460866 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.460848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:28:22.470474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.470451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e20af5-a9a5-464f-81d1-67a8c17db4cf-metrics-certs\") pod \"network-metrics-daemon-jqdbf\" (UID: \"68e20af5-a9a5-464f-81d1-67a8c17db4cf\") " pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:28:22.613562 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.613493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fkx7\""
Jul 17 21:28:22.620656 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.620634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7k8fl\""
Jul 17 21:28:22.621607 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.621582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqdbf"
Jul 17 21:28:22.625000 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.624985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dz78x"
Jul 17 21:28:22.629340 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.629322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:28:22.776234 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.776197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dz78x"]
Jul 17 21:28:22.779305 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:22.779274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a956c0c_f929_4b1c_a9e1_7449f408c8a7.slice/crio-e0dd6ce4a7e8bc740d16906f5d01bcd7a71a0fc37a0873d16260ea6840199af3 WatchSource:0}: Error finding container e0dd6ce4a7e8bc740d16906f5d01bcd7a71a0fc37a0873d16260ea6840199af3: Status 404 returned error can't find the container with id e0dd6ce4a7e8bc740d16906f5d01bcd7a71a0fc37a0873d16260ea6840199af3
Jul 17 21:28:22.789726 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.789699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jqdbf"]
Jul 17 21:28:22.792561 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:22.792532    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e20af5_a9a5_464f_81d1_67a8c17db4cf.slice/crio-060ae3f3668ac32e6c42239bb35f7bc2d946f79b28a7c42a4032a3e117e174a9 WatchSource:0}: Error finding container 060ae3f3668ac32e6c42239bb35f7bc2d946f79b28a7c42a4032a3e117e174a9: Status 404 returned error can't find the container with id 060ae3f3668ac32e6c42239bb35f7bc2d946f79b28a7c42a4032a3e117e174a9
Jul 17 21:28:22.805478 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.805338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4nctl"]
Jul 17 21:28:22.808062 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:22.808039    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3841cca9_8ae0_4673_8009_13230454aa9b.slice/crio-028adf29b47b2dd880c89117a00e8496b733cfd924ce6b4c47f0cfde404aec34 WatchSource:0}: Error finding container 028adf29b47b2dd880c89117a00e8496b733cfd924ce6b4c47f0cfde404aec34: Status 404 returned error can't find the container with id 028adf29b47b2dd880c89117a00e8496b733cfd924ce6b4c47f0cfde404aec34
Jul 17 21:28:22.877445 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.877387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4nctl" event={"ID":"3841cca9-8ae0-4673-8009-13230454aa9b","Type":"ContainerStarted","Data":"f7cf7c504bbc49e78e9b20c51cad50cfb4ed79127aa041e1a1e85633bfe8c430"}
Jul 17 21:28:22.877445 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.877433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4nctl" event={"ID":"3841cca9-8ae0-4673-8009-13230454aa9b","Type":"ContainerStarted","Data":"028adf29b47b2dd880c89117a00e8496b733cfd924ce6b4c47f0cfde404aec34"}
Jul 17 21:28:22.877637 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.877486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:28:22.878613 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.878576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqdbf" event={"ID":"68e20af5-a9a5-464f-81d1-67a8c17db4cf","Type":"ContainerStarted","Data":"060ae3f3668ac32e6c42239bb35f7bc2d946f79b28a7c42a4032a3e117e174a9"}
Jul 17 21:28:22.879517 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.879496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dz78x" event={"ID":"0a956c0c-f929-4b1c-a9e1-7449f408c8a7","Type":"ContainerStarted","Data":"e0dd6ce4a7e8bc740d16906f5d01bcd7a71a0fc37a0873d16260ea6840199af3"}
Jul 17 21:28:22.893243 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:22.893201    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4nctl" podStartSLOduration=65.893185922 podStartE2EDuration="1m5.893185922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:22.892500668 +0000 UTC m=+66.804740959" watchObservedRunningTime="2026-07-17 21:28:22.893185922 +0000 UTC m=+66.805426211"
Jul 17 21:28:23.869702 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.869652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:28:23.870071 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.869715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:28:23.870071 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.869791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:28:23.870071 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.869824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:28:23.872639 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.872617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/534d8a6e-9bad-4091-b341-10bc20d9dd6e-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wvcgz\" (UID: \"534d8a6e-9bad-4091-b341-10bc20d9dd6e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:28:23.872757 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.872688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d9fef3e-0416-41cd-8c83-831674459a8c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5njn5\" (UID: \"8d9fef3e-0416-41cd-8c83-831674459a8c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:28:23.872757 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.872737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9d5559-38df-4b36-90d6-3c572fb461d4-metrics-tls\") pod \"dns-default-b6cwm\" (UID: \"5f9d5559-38df-4b36-90d6-3c572fb461d4\") " pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:28:23.873206 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.873168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"image-registry-7465848bb8-f277d\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") " pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:28:23.884213 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:23.884177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqdbf" event={"ID":"68e20af5-a9a5-464f-81d1-67a8c17db4cf","Type":"ContainerStarted","Data":"c0573a713fdb7063774bf77f6b27a759743fe03174feda5b33cd5c6255a63905"}
Jul 17 21:28:24.079122 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.079046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bd8xd\""
Jul 17 21:28:24.086953 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.086936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:28:24.097879 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.097843    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6b9d7\""
Jul 17 21:28:24.104929 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.104903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qt4k4\""
Jul 17 21:28:24.105948 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.105929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"
Jul 17 21:28:24.113694 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.113664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:28:24.126223 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.126091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2n6kt\""
Jul 17 21:28:24.134229 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.134138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"
Jul 17 21:28:24.297285 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.297255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7465848bb8-f277d"]
Jul 17 21:28:24.303334 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:24.303302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522f30fd_2eb0_47e9_a6c1_0ee6e31a893a.slice/crio-77fadf2102a2b81078960fe0a9987a9cd49cc8a8d75680a7f7402a8e7314aaa4 WatchSource:0}: Error finding container 77fadf2102a2b81078960fe0a9987a9cd49cc8a8d75680a7f7402a8e7314aaa4: Status 404 returned error can't find the container with id 77fadf2102a2b81078960fe0a9987a9cd49cc8a8d75680a7f7402a8e7314aaa4
Jul 17 21:28:24.312778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.312003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz"]
Jul 17 21:28:24.326974 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.326947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b6cwm"]
Jul 17 21:28:24.331193 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:24.331164    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9d5559_38df_4b36_90d6_3c572fb461d4.slice/crio-37c04b7e74e71afc3c4ac535ada201ba1d373c83c32b77d4825718fc6849a995 WatchSource:0}: Error finding container 37c04b7e74e71afc3c4ac535ada201ba1d373c83c32b77d4825718fc6849a995: Status 404 returned error can't find the container with id 37c04b7e74e71afc3c4ac535ada201ba1d373c83c32b77d4825718fc6849a995
Jul 17 21:28:24.354117 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.354048    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5"]
Jul 17 21:28:24.365692 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:24.365665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9fef3e_0416_41cd_8c83_831674459a8c.slice/crio-d384b6087662e71960b51fbf7a41f91a278c9300579fbdbbd761c729348a7e42 WatchSource:0}: Error finding container d384b6087662e71960b51fbf7a41f91a278c9300579fbdbbd761c729348a7e42: Status 404 returned error can't find the container with id d384b6087662e71960b51fbf7a41f91a278c9300579fbdbbd761c729348a7e42
Jul 17 21:28:24.895794 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.895006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqdbf" event={"ID":"68e20af5-a9a5-464f-81d1-67a8c17db4cf","Type":"ContainerStarted","Data":"a953233edeaf207d7012dcf1d69b6150011f925e0a18135cdde3f310f95d62d7"}
Jul 17 21:28:24.899068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.898980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz" event={"ID":"534d8a6e-9bad-4091-b341-10bc20d9dd6e","Type":"ContainerStarted","Data":"ba9d8233e6e8ac82508e19a95fa95ddbc065f2c2874a3ff1bafb515604d49ae1"}
Jul 17 21:28:24.901668 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.901573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7465848bb8-f277d" event={"ID":"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a","Type":"ContainerStarted","Data":"ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95"}
Jul 17 21:28:24.901668 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.901639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7465848bb8-f277d" event={"ID":"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a","Type":"ContainerStarted","Data":"77fadf2102a2b81078960fe0a9987a9cd49cc8a8d75680a7f7402a8e7314aaa4"}
Jul 17 21:28:24.901848 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.901700    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:28:24.904093 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.903994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5" event={"ID":"8d9fef3e-0416-41cd-8c83-831674459a8c","Type":"ContainerStarted","Data":"d384b6087662e71960b51fbf7a41f91a278c9300579fbdbbd761c729348a7e42"}
Jul 17 21:28:24.905917 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.905852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b6cwm" event={"ID":"5f9d5559-38df-4b36-90d6-3c572fb461d4","Type":"ContainerStarted","Data":"37c04b7e74e71afc3c4ac535ada201ba1d373c83c32b77d4825718fc6849a995"}
Jul 17 21:28:24.932136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.932083    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7465848bb8-f277d" podStartSLOduration=66.932063745 podStartE2EDuration="1m6.932063745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:24.931654916 +0000 UTC m=+68.843895207" watchObservedRunningTime="2026-07-17 21:28:24.932063745 +0000 UTC m=+68.844304035"
Jul 17 21:28:24.933589 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:24.932722    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jqdbf" podStartSLOduration=66.980713594 podStartE2EDuration="1m7.932707736s" totalImagesPullingTime="951.994142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:24.913523028 +0000 UTC m=+68.825763345" watchObservedRunningTime="2026-07-17 21:28:24.932707736 +0000 UTC m=+68.844948025"
Jul 17 21:28:26.244397 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.244138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"]
Jul 17 21:28:26.247568 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.247542    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-62d22"]
Jul 17 21:28:26.247809 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.247696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.250967 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.250928    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"]
Jul 17 21:28:26.251401 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.251382    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 21:28:26.252785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.251648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-79p7h\""
Jul 17 21:28:26.252785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.251950    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 21:28:26.252785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.252236    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 21:28:26.252785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.252449    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 21:28:26.252785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.252656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.255443    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"]
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.255668    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.255777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x2w6p\""
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.255878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.255984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.256263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 21:28:26.256674 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.256278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.259181 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.259160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 21:28:26.259377 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.259358    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 21:28:26.260573 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.260537    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"]
Jul 17 21:28:26.260701 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.260660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.261943 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.261680    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 21:28:26.262659 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.262537    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 21:28:26.264424 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.264043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"]
Jul 17 21:28:26.264424 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.264136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.264609 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.264536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mc8m7\""
Jul 17 21:28:26.266982 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.266139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 21:28:26.267096 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.267032    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 17 21:28:26.268145 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.267951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 21:28:26.268145 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.267973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 17 21:28:26.268145 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.267957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 21:28:26.268564 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.268508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"]
Jul 17 21:28:26.268564 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.268543    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 17 21:28:26.270798 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.270777    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"]
Jul 17 21:28:26.271927 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.271883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"]
Jul 17 21:28:26.286949 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.281997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-62d22"]
Jul 17 21:28:26.348540 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.347376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c474549d4-twbtw"]
Jul 17 21:28:26.354003 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.353977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.357108 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 17 21:28:26.357362 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357344    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 21:28:26.357531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 17 21:28:26.357713 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fqflc\""
Jul 17 21:28:26.357827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 21:28:26.357887 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 17 21:28:26.357930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.357889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 17 21:28:26.369379 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.369353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c474549d4-twbtw"]
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea756d28-1f16-4d91-ad54-d047cb94f4b0-snapshots\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l97z\" (UniqueName: \"kubernetes.io/projected/ea756d28-1f16-4d91-ad54-d047cb94f4b0-kube-api-access-2l97z\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea756d28-1f16-4d91-ad54-d047cb94f4b0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6b5aa2e0-8731-40b7-96dc-62212e75ea57-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnhc\" (UniqueName: \"kubernetes.io/projected/6b5aa2e0-8731-40b7-96dc-62212e75ea57-kube-api-access-pjnhc\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea756d28-1f16-4d91-ad54-d047cb94f4b0-tmp\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.392819 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea756d28-1f16-4d91-ad54-d047cb94f4b0-service-ca-bundle\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-ca\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzw2\" (UniqueName: \"kubernetes.io/projected/cba885ce-8051-449d-b2c9-1ba243cbe3d6-kube-api-access-bkzw2\") pod \"managed-serviceaccount-addon-agent-549d76b4fb-z78k8\" (UID: \"cba885ce-8051-449d-b2c9-1ba243cbe3d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.392974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea756d28-1f16-4d91-ad54-d047cb94f4b0-serving-cert\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8dw\" (UniqueName: \"kubernetes.io/projected/6ee8f435-8f90-403c-b956-2359bbc15013-kube-api-access-xj8dw\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-hub\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ee8f435-8f90-403c-b956-2359bbc15013-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cba885ce-8051-449d-b2c9-1ba243cbe3d6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-549d76b4fb-z78k8\" (UID: \"cba885ce-8051-449d-b2c9-1ba243cbe3d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ee8f435-8f90-403c-b956-2359bbc15013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.393292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcn2\" (UniqueName: \"kubernetes.io/projected/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-kube-api-access-kxcn2\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.393776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b5aa2e0-8731-40b7-96dc-62212e75ea57-tmp\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.393776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.393399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.450076 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.450043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-4ztx7"]
Jul 17 21:28:26.453722 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.453501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:26.456164 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.456141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wfmtb\""
Jul 17 21:28:26.456299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.456173    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 21:28:26.456515 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.456492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 21:28:26.467657 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.467632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-4ztx7"]
Jul 17 21:28:26.494519 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.494519 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea756d28-1f16-4d91-ad54-d047cb94f4b0-snapshots\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.494519 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l97z\" (UniqueName: \"kubernetes.io/projected/ea756d28-1f16-4d91-ad54-d047cb94f4b0-kube-api-access-2l97z\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea756d28-1f16-4d91-ad54-d047cb94f4b0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-default-certificate\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-stats-auth\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6b5aa2e0-8731-40b7-96dc-62212e75ea57-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnhc\" (UniqueName: \"kubernetes.io/projected/6b5aa2e0-8731-40b7-96dc-62212e75ea57-kube-api-access-pjnhc\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea756d28-1f16-4d91-ad54-d047cb94f4b0-tmp\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea756d28-1f16-4d91-ad54-d047cb94f4b0-service-ca-bundle\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817083d7-66c5-4f1b-9f4d-d30a0de2607c-service-ca-bundle\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.494822 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-ca\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzw2\" (UniqueName: \"kubernetes.io/projected/cba885ce-8051-449d-b2c9-1ba243cbe3d6-kube-api-access-bkzw2\") pod \"managed-serviceaccount-addon-agent-549d76b4fb-z78k8\" (UID: \"cba885ce-8051-449d-b2c9-1ba243cbe3d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8sm7\" (UniqueName: \"kubernetes.io/projected/817083d7-66c5-4f1b-9f4d-d30a0de2607c-kube-api-access-z8sm7\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea756d28-1f16-4d91-ad54-d047cb94f4b0-serving-cert\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8dw\" (UniqueName: \"kubernetes.io/projected/6ee8f435-8f90-403c-b956-2359bbc15013-kube-api-access-xj8dw\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-hub\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.494988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ee8f435-8f90-403c-b956-2359bbc15013-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cba885ce-8051-449d-b2c9-1ba243cbe3d6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-549d76b4fb-z78k8\" (UID: \"cba885ce-8051-449d-b2c9-1ba243cbe3d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-metrics-certs\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ee8f435-8f90-403c-b956-2359bbc15013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcn2\" (UniqueName: \"kubernetes.io/projected/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-kube-api-access-kxcn2\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b5aa2e0-8731-40b7-96dc-62212e75ea57-tmp\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea756d28-1f16-4d91-ad54-d047cb94f4b0-tmp\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea756d28-1f16-4d91-ad54-d047cb94f4b0-snapshots\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.495345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.496101 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b5aa2e0-8731-40b7-96dc-62212e75ea57-tmp\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.496101 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.495560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea756d28-1f16-4d91-ad54-d047cb94f4b0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.496101 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.496012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea756d28-1f16-4d91-ad54-d047cb94f4b0-service-ca-bundle\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.496258 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.496226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ee8f435-8f90-403c-b956-2359bbc15013-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.499177 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.499133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea756d28-1f16-4d91-ad54-d047cb94f4b0-serving-cert\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.499765 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.499725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ee8f435-8f90-403c-b956-2359bbc15013-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.499932 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.499911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cba885ce-8051-449d-b2c9-1ba243cbe3d6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-549d76b4fb-z78k8\" (UID: \"cba885ce-8051-449d-b2c9-1ba243cbe3d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.500685 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.500643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-ca\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.501821 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.501250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-hub\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.501821 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.501320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.501821 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.501787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.504748 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.504726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6b5aa2e0-8731-40b7-96dc-62212e75ea57-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.508829 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.508797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzw2\" (UniqueName: \"kubernetes.io/projected/cba885ce-8051-449d-b2c9-1ba243cbe3d6-kube-api-access-bkzw2\") pod \"managed-serviceaccount-addon-agent-549d76b4fb-z78k8\" (UID: \"cba885ce-8051-449d-b2c9-1ba243cbe3d6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.512416 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.512374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l97z\" (UniqueName: \"kubernetes.io/projected/ea756d28-1f16-4d91-ad54-d047cb94f4b0-kube-api-access-2l97z\") pod \"insights-operator-555d9d7885-62d22\" (UID: \"ea756d28-1f16-4d91-ad54-d047cb94f4b0\") " pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.512416 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.512385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcn2\" (UniqueName: \"kubernetes.io/projected/77d6e6c3-7ab9-4737-a3ec-300345ae2c3d-kube-api-access-kxcn2\") pod \"cluster-proxy-proxy-agent-756f6c8f4f-7vdp4\" (UID: \"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.514252 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.514231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8dw\" (UniqueName: \"kubernetes.io/projected/6ee8f435-8f90-403c-b956-2359bbc15013-kube-api-access-xj8dw\") pod \"cluster-monitoring-operator-76cf8966f8-72h96\" (UID: \"6ee8f435-8f90-403c-b956-2359bbc15013\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.525035 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.525007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnhc\" (UniqueName: \"kubernetes.io/projected/6b5aa2e0-8731-40b7-96dc-62212e75ea57-kube-api-access-pjnhc\") pod \"klusterlet-addon-workmgr-6c9d7b8dd5-94vhh\" (UID: \"6b5aa2e0-8731-40b7-96dc-62212e75ea57\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.574286 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.574251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"
Jul 17 21:28:26.590488 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.590463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-62d22"
Jul 17 21:28:26.595592 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.595567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-metrics-certs\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.595729 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.595660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tzc\" (UniqueName: \"kubernetes.io/projected/c2081059-6078-4527-9218-fe64c24b2624-kube-api-access-67tzc\") pod \"downloads-5bc67dcd46-4ztx7\" (UID: \"c2081059-6078-4527-9218-fe64c24b2624\") " pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:26.595729 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.595703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-default-certificate\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.595851 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.595729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-stats-auth\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.595851 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.595766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817083d7-66c5-4f1b-9f4d-d30a0de2607c-service-ca-bundle\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.595851 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.595807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8sm7\" (UniqueName: \"kubernetes.io/projected/817083d7-66c5-4f1b-9f4d-d30a0de2607c-kube-api-access-z8sm7\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.597866 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.597839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817083d7-66c5-4f1b-9f4d-d30a0de2607c-service-ca-bundle\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.598065 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.597990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"
Jul 17 21:28:26.598895 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.598854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-metrics-certs\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.599244 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.599224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-stats-auth\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.599641 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.599591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/817083d7-66c5-4f1b-9f4d-d30a0de2607c-default-certificate\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.613292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.612763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8sm7\" (UniqueName: \"kubernetes.io/projected/817083d7-66c5-4f1b-9f4d-d30a0de2607c-kube-api-access-z8sm7\") pod \"router-default-c474549d4-twbtw\" (UID: \"817083d7-66c5-4f1b-9f4d-d30a0de2607c\") " pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.614985 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.614964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"
Jul 17 21:28:26.622314 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.622293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:26.666278 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.666246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:26.696235 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.696193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tzc\" (UniqueName: \"kubernetes.io/projected/c2081059-6078-4527-9218-fe64c24b2624-kube-api-access-67tzc\") pod \"downloads-5bc67dcd46-4ztx7\" (UID: \"c2081059-6078-4527-9218-fe64c24b2624\") " pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:26.715308 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.715269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tzc\" (UniqueName: \"kubernetes.io/projected/c2081059-6078-4527-9218-fe64c24b2624-kube-api-access-67tzc\") pod \"downloads-5bc67dcd46-4ztx7\" (UID: \"c2081059-6078-4527-9218-fe64c24b2624\") " pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:26.765343 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:26.765258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:28.572429 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.565163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4"]
Jul 17 21:28:28.583984 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.583882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96"]
Jul 17 21:28:28.619215 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:28.599632    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77d6e6c3_7ab9_4737_a3ec_300345ae2c3d.slice/crio-9512cb6d54800609d40f182a51fcf2e6bd7fc6181690be6cdccbc14f1f278b5a WatchSource:0}: Error finding container 9512cb6d54800609d40f182a51fcf2e6bd7fc6181690be6cdccbc14f1f278b5a: Status 404 returned error can't find the container with id 9512cb6d54800609d40f182a51fcf2e6bd7fc6181690be6cdccbc14f1f278b5a
Jul 17 21:28:28.620068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.620028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8"]
Jul 17 21:28:28.630774 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.630422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c474549d4-twbtw"]
Jul 17 21:28:28.644715 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.644622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"]
Jul 17 21:28:28.649924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.649796    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-4ztx7"]
Jul 17 21:28:28.676250 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.672878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-62d22"]
Jul 17 21:28:28.686769 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:28.686737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea756d28_1f16_4d91_ad54_d047cb94f4b0.slice/crio-e04cc4fddf5dbb7dc1489ecd15f11de4c3e08a4cff5d6d26e25d90119dce80d4 WatchSource:0}: Error finding container e04cc4fddf5dbb7dc1489ecd15f11de4c3e08a4cff5d6d26e25d90119dce80d4: Status 404 returned error can't find the container with id e04cc4fddf5dbb7dc1489ecd15f11de4c3e08a4cff5d6d26e25d90119dce80d4
Jul 17 21:28:28.923174 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.923139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5" event={"ID":"8d9fef3e-0416-41cd-8c83-831674459a8c","Type":"ContainerStarted","Data":"d20a4d886ee80a2ec5e36cb5614cd72c30b0de082eea41f33ba576f34645f774"}
Jul 17 21:28:28.925276 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.925249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b6cwm" event={"ID":"5f9d5559-38df-4b36-90d6-3c572fb461d4","Type":"ContainerStarted","Data":"a3b34e43ee21d16fdd442949d5dccdcbb0186695c7bacec511fcad6bf2f9046e"}
Jul 17 21:28:28.925411 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.925296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b6cwm" event={"ID":"5f9d5559-38df-4b36-90d6-3c572fb461d4","Type":"ContainerStarted","Data":"d41c20cd0264e92b58a77da160f9eefce877f320c43b3bb07cb3a9fb9ffe8d2a"}
Jul 17 21:28:28.925411 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.925344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:28:28.926891 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.926866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c474549d4-twbtw" event={"ID":"817083d7-66c5-4f1b-9f4d-d30a0de2607c","Type":"ContainerStarted","Data":"ceb2daf736f6f57605de8c153003ec3fff2f505ecb7e8874c002237a61fdd355"}
Jul 17 21:28:28.926991 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.926909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c474549d4-twbtw" event={"ID":"817083d7-66c5-4f1b-9f4d-d30a0de2607c","Type":"ContainerStarted","Data":"d3750bfbf2837b5c4e54334b1a3e468870a3d9577a6ab5a83b8c79a580790fb8"}
Jul 17 21:28:28.928342 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.928317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dz78x" event={"ID":"0a956c0c-f929-4b1c-a9e1-7449f408c8a7","Type":"ContainerStarted","Data":"27f0590e978d8f93d011bcf946a8eb886a605bf3fedd17033a10aeb6b0700434"}
Jul 17 21:28:28.929529 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.929504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-4ztx7" event={"ID":"c2081059-6078-4527-9218-fe64c24b2624","Type":"ContainerStarted","Data":"00170c014b426e52b4725d68f71ec2047fc86c22cb475fe6c3937153e3b07b66"}
Jul 17 21:28:28.930665 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.930642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4" event={"ID":"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d","Type":"ContainerStarted","Data":"9512cb6d54800609d40f182a51fcf2e6bd7fc6181690be6cdccbc14f1f278b5a"}
Jul 17 21:28:28.931782 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.931762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96" event={"ID":"6ee8f435-8f90-403c-b956-2359bbc15013","Type":"ContainerStarted","Data":"877da3937ff52972eeae11600c220a3c72e95c7fa05c8451749eff19979e0ef6"}
Jul 17 21:28:28.933359 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.933341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz" event={"ID":"534d8a6e-9bad-4091-b341-10bc20d9dd6e","Type":"ContainerStarted","Data":"2f6858e5044638806382ed26d84c6dca3509d2322d0422652b5ebff9612f12e7"}
Jul 17 21:28:28.933451 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.933373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz" event={"ID":"534d8a6e-9bad-4091-b341-10bc20d9dd6e","Type":"ContainerStarted","Data":"803a02f9d7363b27bd372ca93a487dd8523cebe068017ed4794af244dcef8bf5"}
Jul 17 21:28:28.934460 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.934440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh" event={"ID":"6b5aa2e0-8731-40b7-96dc-62212e75ea57","Type":"ContainerStarted","Data":"d30853af83048b894a32b26827cce6efc91fcbcf2582dac50e3a8534de31e108"}
Jul 17 21:28:28.935540 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.935521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-62d22" event={"ID":"ea756d28-1f16-4d91-ad54-d047cb94f4b0","Type":"ContainerStarted","Data":"e04cc4fddf5dbb7dc1489ecd15f11de4c3e08a4cff5d6d26e25d90119dce80d4"}
Jul 17 21:28:28.936587 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.936565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8" event={"ID":"cba885ce-8051-449d-b2c9-1ba243cbe3d6","Type":"ContainerStarted","Data":"343951f614540dc66a659370b8654912fc8465c136763446c6658ba4e203651d"}
Jul 17 21:28:28.939554 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.939508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5njn5" podStartSLOduration=34.035355415 podStartE2EDuration="37.939494545s" totalImagesPullingTime="3.90413913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:28.938752963 +0000 UTC m=+72.850993252" watchObservedRunningTime="2026-07-17 21:28:28.939494545 +0000 UTC m=+72.851734833"
Jul 17 21:28:28.956722 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.956659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wvcgz" podStartSLOduration=66.041922833 podStartE2EDuration="1m9.956639366s" totalImagesPullingTime="3.914716533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:28.954807369 +0000 UTC m=+72.867047658" watchObservedRunningTime="2026-07-17 21:28:28.956639366 +0000 UTC m=+72.868879656"
Jul 17 21:28:28.975923 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:28.975847    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dz78x" podStartSLOduration=66.38297867 podStartE2EDuration="1m11.975830186s" totalImagesPullingTime="5.592851516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:28.97505781 +0000 UTC m=+72.887298100" watchObservedRunningTime="2026-07-17 21:28:28.975830186 +0000 UTC m=+72.888070455"
Jul 17 21:28:29.011039 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.010987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c474549d4-twbtw" podStartSLOduration=3.010971353 podStartE2EDuration="3.010971353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:29.01043194 +0000 UTC m=+72.922672230" watchObservedRunningTime="2026-07-17 21:28:29.010971353 +0000 UTC m=+72.923211623"
Jul 17 21:28:29.011469 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.011446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b6cwm" podStartSLOduration=34.07481541 podStartE2EDuration="38.011437839s" totalImagesPullingTime="3.936622429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:28.992513026 +0000 UTC m=+72.904753316" watchObservedRunningTime="2026-07-17 21:28:29.011437839 +0000 UTC m=+72.923678164"
Jul 17 21:28:29.206889 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.206856    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c547d8b76-779c4"]
Jul 17 21:28:29.218299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.218267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.219355 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.219330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c547d8b76-779c4"]
Jul 17 21:28:29.220561 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.220535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 21:28:29.220776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.220754    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cnsv4\""
Jul 17 21:28:29.220860 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.220815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 21:28:29.221068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.221049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 21:28:29.221194 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.221115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 21:28:29.221282 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.221194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 21:28:29.318497 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.318457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-config\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.318697 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.318506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-oauth-serving-cert\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.318697 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.318620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zjn\" (UniqueName: \"kubernetes.io/projected/bb3371b1-d37d-44cd-8945-e1c7a022306a-kube-api-access-75zjn\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.318697 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.318679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-oauth-config\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.318868 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.318712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-service-ca\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.318868 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.318752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-serving-cert\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.419592 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.419554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-config\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.419781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.419635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-oauth-serving-cert\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.419781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.419662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zjn\" (UniqueName: \"kubernetes.io/projected/bb3371b1-d37d-44cd-8945-e1c7a022306a-kube-api-access-75zjn\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.419781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.419697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-oauth-config\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.419781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.419724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-service-ca\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.419781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.419757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-serving-cert\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.420914 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.420879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-service-ca\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.421070 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.421043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-config\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.421212 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.421190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-oauth-serving-cert\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.423241 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.423221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-serving-cert\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.423387 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.423347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-oauth-config\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.430522 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.430500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zjn\" (UniqueName: \"kubernetes.io/projected/bb3371b1-d37d-44cd-8945-e1c7a022306a-kube-api-access-75zjn\") pod \"console-7c547d8b76-779c4\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") " pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.528492 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.528414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:29.667580 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.667548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:29.671167 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.671139    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:29.749426 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.749211    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c547d8b76-779c4"]
Jul 17 21:28:29.946721 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.946430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c547d8b76-779c4" event={"ID":"bb3371b1-d37d-44cd-8945-e1c7a022306a","Type":"ContainerStarted","Data":"2a0f270d1c40a8d0279ce1b30fdea9daf40efcfb9d2d25c0fbbf35b050ef9d6d"}
Jul 17 21:28:29.948260 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.947639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:29.949353 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:29.949148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c474549d4-twbtw"
Jul 17 21:28:38.950527 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:38.950492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b6cwm"
Jul 17 21:28:39.955354 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:39.953820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bk6gm"]
Jul 17 21:28:39.956776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:39.956735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:39.960955 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:39.960888    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn2hx\""
Jul 17 21:28:39.961159 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:39.961123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 21:28:39.961363 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:39.961347    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 21:28:39.972879 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:39.972193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bk6gm"]
Jul 17 21:28:40.018616 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.018569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4" event={"ID":"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d","Type":"ContainerStarted","Data":"cd4a5322f487a60b5509a7d05939a2adc2d6df5186e8577d02c81ddf28d8698a"}
Jul 17 21:28:40.018769 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.018635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4" event={"ID":"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d","Type":"ContainerStarted","Data":"4c65c8253c80ad34ebbb4069d6a40eba5353dbe6c75b1fda80e1674669a41d4f"}
Jul 17 21:28:40.018769 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.018652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4" event={"ID":"77d6e6c3-7ab9-4737-a3ec-300345ae2c3d","Type":"ContainerStarted","Data":"e4f0b7346b19fa35f885c074be8fbebe74ef9ecc6ef6f2a77d6112e5440b785f"}
Jul 17 21:28:40.021327 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.021298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96" event={"ID":"6ee8f435-8f90-403c-b956-2359bbc15013","Type":"ContainerStarted","Data":"cf84ef8d6631b2518e32882dbf8dccd30abaf3bf38034067ecc5f29ae313acce"}
Jul 17 21:28:40.023857 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.023829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh" event={"ID":"6b5aa2e0-8731-40b7-96dc-62212e75ea57","Type":"ContainerStarted","Data":"0bc660647cf114eacdf58905e549800225c7b4d8548e639767095dcfd42e7efc"}
Jul 17 21:28:40.024561 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.024538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:40.026813 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.026791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh"
Jul 17 21:28:40.027216 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.027190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-62d22" event={"ID":"ea756d28-1f16-4d91-ad54-d047cb94f4b0","Type":"ContainerStarted","Data":"49367fbb8e661a35c73e84bae6d56fb89815d2fa9ec28f0ca8b2431fb806ed21"}
Jul 17 21:28:40.030645 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.030580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8" event={"ID":"cba885ce-8051-449d-b2c9-1ba243cbe3d6","Type":"ContainerStarted","Data":"dbe20ceaa38c620b36b6c29cee2eb102db08f631908b4e442c374f537b9242d5"}
Jul 17 21:28:40.033613 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.033541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c547d8b76-779c4" event={"ID":"bb3371b1-d37d-44cd-8945-e1c7a022306a","Type":"ContainerStarted","Data":"cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7"}
Jul 17 21:28:40.054108 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.054055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-756f6c8f4f-7vdp4" podStartSLOduration=3.299681562 podStartE2EDuration="14.054032438s" totalImagesPullingTime="10.754350876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:40.052145044 +0000 UTC m=+83.964385354" watchObservedRunningTime="2026-07-17 21:28:40.054032438 +0000 UTC m=+83.966272731"
Jul 17 21:28:40.078994 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.078932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-72h96" podStartSLOduration=3.350053015 podStartE2EDuration="14.078910382s" totalImagesPullingTime="10.728857367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:40.07715476 +0000 UTC m=+83.989395070" watchObservedRunningTime="2026-07-17 21:28:40.078910382 +0000 UTC m=+83.991150673"
Jul 17 21:28:40.101856 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.101623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c9d7b8dd5-94vhh" podStartSLOduration=3.373240485 podStartE2EDuration="14.101587247s" totalImagesPullingTime="10.728346762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:40.101006954 +0000 UTC m=+84.013247244" watchObservedRunningTime="2026-07-17 21:28:40.101587247 +0000 UTC m=+84.013827536"
Jul 17 21:28:40.116224 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.116184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf151b43-656f-4727-b3cf-2b97d0c3d045-data-volume\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.116407 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.116373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4bb\" (UniqueName: \"kubernetes.io/projected/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-api-access-zt4bb\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.116481 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.116468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.116558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.116542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf151b43-656f-4727-b3cf-2b97d0c3d045-crio-socket\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.116647 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.116579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.121936 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.121638    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c547d8b76-779c4" podStartSLOduration=1.527149472 podStartE2EDuration="11.121620609s" totalImagesPullingTime="9.594471137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:40.120990415 +0000 UTC m=+84.033230724" watchObservedRunningTime="2026-07-17 21:28:40.121620609 +0000 UTC m=+84.033860890"
Jul 17 21:28:40.137379 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.137329    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-549d76b4fb-z78k8" podStartSLOduration=3.364288039 podStartE2EDuration="14.137312057s" totalImagesPullingTime="10.773024018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:40.13606738 +0000 UTC m=+84.048307662" watchObservedRunningTime="2026-07-17 21:28:40.137312057 +0000 UTC m=+84.049552345"
Jul 17 21:28:40.156623 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.156535    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-62d22" podStartSLOduration=3.497991426 podStartE2EDuration="14.156521028s" totalImagesPullingTime="10.658529602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:40.155034422 +0000 UTC m=+84.067274733" watchObservedRunningTime="2026-07-17 21:28:40.156521028 +0000 UTC m=+84.068761318"
Jul 17 21:28:40.217761 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.217686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf151b43-656f-4727-b3cf-2b97d0c3d045-data-volume\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.217885 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.217786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4bb\" (UniqueName: \"kubernetes.io/projected/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-api-access-zt4bb\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.217885 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.217840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.217968 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.217885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf151b43-656f-4727-b3cf-2b97d0c3d045-crio-socket\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.217968 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.217907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.218033 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.218026    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:40.218090 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.218078    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:40.718059312 +0000 UTC m=+84.630299579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:40.218552 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.218536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf151b43-656f-4727-b3cf-2b97d0c3d045-data-volume\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.219358 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.219049    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:28:40.219358 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.219058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf151b43-656f-4727-b3cf-2b97d0c3d045-crio-socket\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.219358 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.219116    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:40.719099443 +0000 UTC m=+84.631339721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : configmap "kube-rbac-proxy" not found
Jul 17 21:28:40.228069 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.228041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4bb\" (UniqueName: \"kubernetes.io/projected/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-api-access-zt4bb\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.724134 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.724093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.724333 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:40.724151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:40.724333 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.724251    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:28:40.724333 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.724285    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:40.724502 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.724337    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:41.724313534 +0000 UTC m=+85.636553804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : configmap "kube-rbac-proxy" not found
Jul 17 21:28:40.724502 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:40.724357    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:41.724347663 +0000 UTC m=+85.636587938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:41.734213 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:41.734172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:41.734719 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:41.734246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:41.734719 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:41.734308    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:28:41.734719 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:41.734398    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:43.73437694 +0000 UTC m=+87.646617225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : configmap "kube-rbac-proxy" not found
Jul 17 21:28:41.734719 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:41.734408    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:41.734719 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:41.734459    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:43.734445254 +0000 UTC m=+87.646685545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:41.750068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:41.750039    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-5njn5_8d9fef3e-0416-41cd-8c83-831674459a8c/networking-console-plugin/0.log"
Jul 17 21:28:41.949437 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:41.949331    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:28:42.153109 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:42.153032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/2.log"
Jul 17 21:28:42.550496 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:42.550463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c547d8b76-779c4_bb3371b1-d37d-44cd-8945-e1c7a022306a/console/0.log"
Jul 17 21:28:42.949657 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:42.949623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b6cwm_5f9d5559-38df-4b36-90d6-3c572fb461d4/dns/0.log"
Jul 17 21:28:43.150384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:43.150333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b6cwm_5f9d5559-38df-4b36-90d6-3c572fb461d4/kube-rbac-proxy/0.log"
Jul 17 21:28:43.657366 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:43.657331    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c547d8b76-779c4"]
Jul 17 21:28:43.754998 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:43.754950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:43.755180 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:43.755028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:43.755180 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:43.755082    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:28:43.755180 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:43.755159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.75513727 +0000 UTC m=+91.667377558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : configmap "kube-rbac-proxy" not found
Jul 17 21:28:43.755330 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:43.755189    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:43.755330 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:43.755252    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.755231878 +0000 UTC m=+91.667472160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:44.092797 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:44.092764    2563 patch_prober.go:28] interesting pod/image-registry-7465848bb8-f277d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 21:28:44.093197 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:44.092826    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7465848bb8-f277d" podUID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:28:44.549776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:44.549744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97sf8_a05c180a-04d3-4fa2-904d-034acaf5ada9/dns-node-resolver/0.log"
Jul 17 21:28:44.950826 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:44.950785    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7465848bb8-f277d_522f30fd-2eb0-47e9-a6c1-0ee6e31a893a/registry/0.log"
Jul 17 21:28:45.549941 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:45.549916    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9c7k_4fdc8bd8-1135-46ad-a9e9-4e06928fc07e/node-ca/0.log"
Jul 17 21:28:45.750187 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:45.750123    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c474549d4-twbtw_817083d7-66c5-4f1b-9f4d-d30a0de2607c/router/0.log"
Jul 17 21:28:45.915532 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:45.915509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:28:46.354561 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.354481    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-psfq2_600ae2a5-5709-473f-b2c1-63145e434e91/serve-healthcheck-canary/0.log"
Jul 17 21:28:46.546635 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.545122    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dfchh"]
Jul 17 21:28:46.591113 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.591070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.595145 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.595114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 21:28:46.595282 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.595166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 21:28:46.595282 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.595281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cbqw8\""
Jul 17 21:28:46.595393 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.595327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 21:28:46.595463 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.595447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 21:28:46.681128 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d165ff68-206a-451e-b307-bfcc5c190313-metrics-client-ca\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-sys\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-textfile\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqbs\" (UniqueName: \"kubernetes.io/projected/d165ff68-206a-451e-b307-bfcc5c190313-kube-api-access-drqbs\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681483 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-root\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681483 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-wtmp\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681483 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-accelerators-collector-config\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.681629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.681545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-tls\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-accelerators-collector-config\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-tls\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d165ff68-206a-451e-b307-bfcc5c190313-metrics-client-ca\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-sys\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-textfile\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqbs\" (UniqueName: \"kubernetes.io/projected/d165ff68-206a-451e-b307-bfcc5c190313-kube-api-access-drqbs\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-root\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-wtmp\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.782617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.783204 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.782685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-sys\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.783204 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.783097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-accelerators-collector-config\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.789440 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.783810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d165ff68-206a-451e-b307-bfcc5c190313-metrics-client-ca\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.789440 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.783870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-root\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.789440 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.783980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-wtmp\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.789440 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.784034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-textfile\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.789440 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.789214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.790413 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.790390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d165ff68-206a-451e-b307-bfcc5c190313-node-exporter-tls\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.793919 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.793894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqbs\" (UniqueName: \"kubernetes.io/projected/d165ff68-206a-451e-b307-bfcc5c190313-kube-api-access-drqbs\") pod \"node-exporter-dfchh\" (UID: \"d165ff68-206a-451e-b307-bfcc5c190313\") " pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:46.904811 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:46.904661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dfchh"
Jul 17 21:28:47.792125 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:47.792085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:47.792635 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:47.792206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:47.792635 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:47.792230    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:47.792635 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:47.792299    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:28:47.792635 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:47.792318    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:55.792294163 +0000 UTC m=+99.704534442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:28:47.792635 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:28:47.792355    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm podName:cf151b43-656f-4727-b3cf-2b97d0c3d045 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:55.792340203 +0000 UTC m=+99.704580488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bk6gm" (UID: "cf151b43-656f-4727-b3cf-2b97d0c3d045") : configmap "kube-rbac-proxy" not found
Jul 17 21:28:49.528871 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:49.528831    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:28:50.272153 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:50.272115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd165ff68_206a_451e_b307_bfcc5c190313.slice/crio-e102992c11b9c8404e3066500c2b921877b40ac384c39c379f9ab2b6ed391d48 WatchSource:0}: Error finding container e102992c11b9c8404e3066500c2b921877b40ac384c39c379f9ab2b6ed391d48: Status 404 returned error can't find the container with id e102992c11b9c8404e3066500c2b921877b40ac384c39c379f9ab2b6ed391d48
Jul 17 21:28:51.088367 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:51.088291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dfchh" event={"ID":"d165ff68-206a-451e-b307-bfcc5c190313","Type":"ContainerStarted","Data":"e102992c11b9c8404e3066500c2b921877b40ac384c39c379f9ab2b6ed391d48"}
Jul 17 21:28:51.091115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:51.089911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-4ztx7" event={"ID":"c2081059-6078-4527-9218-fe64c24b2624","Type":"ContainerStarted","Data":"9756578a7bff6a50db3fd4f23e4d5ddfc868ff5da33ae44661754339a56ac8c0"}
Jul 17 21:28:51.091115 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:51.091054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:51.104274 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:51.104219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-4ztx7"
Jul 17 21:28:51.108952 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:51.108897    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-4ztx7" podStartSLOduration=3.118081181 podStartE2EDuration="25.108876142s" totalImagesPullingTime="21.990794961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:51.107733279 +0000 UTC m=+95.019973589" watchObservedRunningTime="2026-07-17 21:28:51.108876142 +0000 UTC m=+95.021116432"
Jul 17 21:28:52.096914 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:52.096880    2563 generic.go:356] "Generic (PLEG): container finished" podID="d165ff68-206a-451e-b307-bfcc5c190313" containerID="3648ce5c289ec71939168a58fc09609c82e340b4330fa12804ac299bb113ca83" exitCode=0
Jul 17 21:28:52.097403 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:52.096970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dfchh" event={"ID":"d165ff68-206a-451e-b307-bfcc5c190313","Type":"ContainerDied","Data":"3648ce5c289ec71939168a58fc09609c82e340b4330fa12804ac299bb113ca83"}
Jul 17 21:28:53.104243 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:53.104187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dfchh" event={"ID":"d165ff68-206a-451e-b307-bfcc5c190313","Type":"ContainerStarted","Data":"533471b4bb9f2f39700c693185adcf73c6113a210d80d1acc84e50779a5b9b68"}
Jul 17 21:28:53.104737 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:53.104654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dfchh" event={"ID":"d165ff68-206a-451e-b307-bfcc5c190313","Type":"ContainerStarted","Data":"15a02749f1640adfa9ceb82c450b5a07746701c7418cf2a631f24da114a7a39f"}
Jul 17 21:28:53.127690 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:53.127637    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dfchh" podStartSLOduration=5.572057152 podStartE2EDuration="7.127621088s" totalImagesPullingTime="1.445557755s" totalInitContainerRuntime="110.006181ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:53.12596219 +0000 UTC m=+97.038202478" watchObservedRunningTime="2026-07-17 21:28:53.127621088 +0000 UTC m=+97.039861379"
Jul 17 21:28:53.887337 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:53.887304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4nctl"
Jul 17 21:28:55.411329 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:55.411291    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7465848bb8-f277d"]
Jul 17 21:28:55.873120 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:55.873031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:55.873289 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:55.873138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:55.873806 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:55.873779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf151b43-656f-4727-b3cf-2b97d0c3d045-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:55.875917 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:55.875890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf151b43-656f-4727-b3cf-2b97d0c3d045-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bk6gm\" (UID: \"cf151b43-656f-4727-b3cf-2b97d0c3d045\") " pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:56.175003 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:56.174972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bk6gm"
Jul 17 21:28:56.385276 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:56.385214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bk6gm"]
Jul 17 21:28:56.388042 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:28:56.388005    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf151b43_656f_4727_b3cf_2b97d0c3d045.slice/crio-1e197f9d9244b3e2c2728fb61bf77f01b94e12357668db0a134dad3fdbbc4481 WatchSource:0}: Error finding container 1e197f9d9244b3e2c2728fb61bf77f01b94e12357668db0a134dad3fdbbc4481: Status 404 returned error can't find the container with id 1e197f9d9244b3e2c2728fb61bf77f01b94e12357668db0a134dad3fdbbc4481
Jul 17 21:28:57.127516 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:28:57.127348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bk6gm" event={"ID":"cf151b43-656f-4727-b3cf-2b97d0c3d045","Type":"ContainerStarted","Data":"1e197f9d9244b3e2c2728fb61bf77f01b94e12357668db0a134dad3fdbbc4481"}
Jul 17 21:29:03.157154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:03.157121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bk6gm" event={"ID":"cf151b43-656f-4727-b3cf-2b97d0c3d045","Type":"ContainerStarted","Data":"1083dbd5d3c7bdb31fbf8e69bcd4b146f1633b161d90ea3d3f7e5b321232ec01"}
Jul 17 21:29:08.181467 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.181426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bk6gm" event={"ID":"cf151b43-656f-4727-b3cf-2b97d0c3d045","Type":"ContainerStarted","Data":"98c8767706aac4838eba6123ce23bbce218b75d1ecd209672f80fc16cd2b2f07"}
Jul 17 21:29:08.183214 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.183175    2563 generic.go:356] "Generic (PLEG): container finished" podID="76d7418b-0187-4848-a3e1-8fcbab28dc4b" containerID="2ef7fce8c87b51f4dc4d0d421e7d51f6cef4f2cdc0deae94751143d4ecea4102" exitCode=0
Jul 17 21:29:08.183350 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.183262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6" event={"ID":"76d7418b-0187-4848-a3e1-8fcbab28dc4b","Type":"ContainerDied","Data":"2ef7fce8c87b51f4dc4d0d421e7d51f6cef4f2cdc0deae94751143d4ecea4102"}
Jul 17 21:29:08.183694 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.183664    2563 scope.go:117] "RemoveContainer" containerID="2ef7fce8c87b51f4dc4d0d421e7d51f6cef4f2cdc0deae94751143d4ecea4102"
Jul 17 21:29:08.185022 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.185003    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee5cb13c-da63-472a-816d-71d66eeeeda9" containerID="17a3ebfa932f9fc67f243311e7457d031421e2c7ad928a6757f5315d843d2415" exitCode=0
Jul 17 21:29:08.185107 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.185039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c" event={"ID":"ee5cb13c-da63-472a-816d-71d66eeeeda9","Type":"ContainerDied","Data":"17a3ebfa932f9fc67f243311e7457d031421e2c7ad928a6757f5315d843d2415"}
Jul 17 21:29:08.185326 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.185308    2563 scope.go:117] "RemoveContainer" containerID="17a3ebfa932f9fc67f243311e7457d031421e2c7ad928a6757f5315d843d2415"
Jul 17 21:29:08.682781 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:08.682734    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c547d8b76-779c4" podUID="bb3371b1-d37d-44cd-8945-e1c7a022306a" containerName="console" containerID="cri-o://cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7" gracePeriod=15
Jul 17 21:29:09.007533 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.007500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:29:09.107900 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.107860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-oauth-serving-cert\") pod \"bb3371b1-d37d-44cd-8945-e1c7a022306a\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") "
Jul 17 21:29:09.107900 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.107935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-config\") pod \"bb3371b1-d37d-44cd-8945-e1c7a022306a\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") "
Jul 17 21:29:09.108567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.107993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-service-ca\") pod \"bb3371b1-d37d-44cd-8945-e1c7a022306a\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") "
Jul 17 21:29:09.108567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.108045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75zjn\" (UniqueName: \"kubernetes.io/projected/bb3371b1-d37d-44cd-8945-e1c7a022306a-kube-api-access-75zjn\") pod \"bb3371b1-d37d-44cd-8945-e1c7a022306a\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") "
Jul 17 21:29:09.108567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.108086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-serving-cert\") pod \"bb3371b1-d37d-44cd-8945-e1c7a022306a\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") "
Jul 17 21:29:09.108567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.108116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-oauth-config\") pod \"bb3371b1-d37d-44cd-8945-e1c7a022306a\" (UID: \"bb3371b1-d37d-44cd-8945-e1c7a022306a\") "
Jul 17 21:29:09.108567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.108357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bb3371b1-d37d-44cd-8945-e1c7a022306a" (UID: "bb3371b1-d37d-44cd-8945-e1c7a022306a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:29:09.108567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.108552    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-oauth-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:09.109112 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.109085    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-service-ca" (OuterVolumeSpecName: "service-ca") pod "bb3371b1-d37d-44cd-8945-e1c7a022306a" (UID: "bb3371b1-d37d-44cd-8945-e1c7a022306a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:29:09.109354 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.109321    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-config" (OuterVolumeSpecName: "console-config") pod "bb3371b1-d37d-44cd-8945-e1c7a022306a" (UID: "bb3371b1-d37d-44cd-8945-e1c7a022306a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:29:09.114817 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.114338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3371b1-d37d-44cd-8945-e1c7a022306a-kube-api-access-75zjn" (OuterVolumeSpecName: "kube-api-access-75zjn") pod "bb3371b1-d37d-44cd-8945-e1c7a022306a" (UID: "bb3371b1-d37d-44cd-8945-e1c7a022306a"). InnerVolumeSpecName "kube-api-access-75zjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:29:09.114945 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.114913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bb3371b1-d37d-44cd-8945-e1c7a022306a" (UID: "bb3371b1-d37d-44cd-8945-e1c7a022306a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:29:09.121273 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.120668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bb3371b1-d37d-44cd-8945-e1c7a022306a" (UID: "bb3371b1-d37d-44cd-8945-e1c7a022306a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:29:09.192359 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.192319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-cmc9c" event={"ID":"ee5cb13c-da63-472a-816d-71d66eeeeda9","Type":"ContainerStarted","Data":"c7d8a99ef9a558069f9b142320197ccf8879aa047173a8e37ff5cb2e704def2a"}
Jul 17 21:29:09.194261 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.194001    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb3371b1-d37d-44cd-8945-e1c7a022306a" containerID="cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7" exitCode=0
Jul 17 21:29:09.194261 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.194045    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c547d8b76-779c4"
Jul 17 21:29:09.194261 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.194100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c547d8b76-779c4" event={"ID":"bb3371b1-d37d-44cd-8945-e1c7a022306a","Type":"ContainerDied","Data":"cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7"}
Jul 17 21:29:09.194261 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.194136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c547d8b76-779c4" event={"ID":"bb3371b1-d37d-44cd-8945-e1c7a022306a","Type":"ContainerDied","Data":"2a0f270d1c40a8d0279ce1b30fdea9daf40efcfb9d2d25c0fbbf35b050ef9d6d"}
Jul 17 21:29:09.194261 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.194163    2563 scope.go:117] "RemoveContainer" containerID="cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7"
Jul 17 21:29:09.197507 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.197484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lsnt6" event={"ID":"76d7418b-0187-4848-a3e1-8fcbab28dc4b","Type":"ContainerStarted","Data":"b1b73b3de733ccfcf9a06c490b159e8a660c1c60ea561b02ea3c1533b7b940c5"}
Jul 17 21:29:09.206415 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.206397    2563 scope.go:117] "RemoveContainer" containerID="cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7"
Jul 17 21:29:09.206724 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:29:09.206702    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7\": container with ID starting with cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7 not found: ID does not exist" containerID="cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7"
Jul 17 21:29:09.206807 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.206732    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7"} err="failed to get container status \"cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7\": rpc error: code = NotFound desc = could not find container \"cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7\": container with ID starting with cf3bc920a4bee7b6fb33bf9585a3ddffbd31b095fe5ab9a6f2b0ba3356d8d3c7 not found: ID does not exist"
Jul 17 21:29:09.209354 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.209334    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:09.209447 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.209357    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3371b1-d37d-44cd-8945-e1c7a022306a-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:09.209447 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.209371    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zjn\" (UniqueName: \"kubernetes.io/projected/bb3371b1-d37d-44cd-8945-e1c7a022306a-kube-api-access-75zjn\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:09.209447 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.209385    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:09.209447 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.209398    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb3371b1-d37d-44cd-8945-e1c7a022306a-console-oauth-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:09.251154 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.251043    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c547d8b76-779c4"]
Jul 17 21:29:09.257383 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:09.257356    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c547d8b76-779c4"]
Jul 17 21:29:10.603448 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:10.603416    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3371b1-d37d-44cd-8945-e1c7a022306a" path="/var/lib/kubelet/pods/bb3371b1-d37d-44cd-8945-e1c7a022306a/volumes"
Jul 17 21:29:12.214310 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:12.214208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bk6gm" event={"ID":"cf151b43-656f-4727-b3cf-2b97d0c3d045","Type":"ContainerStarted","Data":"73871cfa1028491940e411684322f258c267d1747ffab0dd2e3c295822fec29c"}
Jul 17 21:29:12.234652 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:12.234570    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bk6gm" podStartSLOduration=17.9118909 podStartE2EDuration="33.234553469s" totalImagesPullingTime="15.322662569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:29:12.232726814 +0000 UTC m=+116.144967103" watchObservedRunningTime="2026-07-17 21:29:12.234553469 +0000 UTC m=+116.146793758"
Jul 17 21:29:20.437970 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.437913    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7465848bb8-f277d" podUID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" containerName="registry" containerID="cri-o://ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95" gracePeriod=30
Jul 17 21:29:20.716561 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.716488    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:29:20.909157 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909126    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-bound-sa-token\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909175    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-installation-pull-secrets\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-certificates\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-ca-trust-extracted\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr59f\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-kube-api-access-tr59f\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909335 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-image-registry-private-configuration\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909583 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-trusted-ca\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909583 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") pod \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\" (UID: \"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a\") "
Jul 17 21:29:20.909751 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.909727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:29:20.910421 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.910377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:29:20.912102 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.912060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:29:20.912102 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.912065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:29:20.912267 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.912162    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:29:20.912448 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.912429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-kube-api-access-tr59f" (OuterVolumeSpecName: "kube-api-access-tr59f") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "kube-api-access-tr59f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:29:20.912518 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.912501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:29:20.918230 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:20.918206    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" (UID: "522f30fd-2eb0-47e9-a6c1-0ee6e31a893a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:29:21.011628 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.011080    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-certificates\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.011832 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.011813    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-ca-trust-extracted\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.011942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.011930    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr59f\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-kube-api-access-tr59f\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.012024 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.012012    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-image-registry-private-configuration\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.012101 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.012090    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-trusted-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.012174 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.012164    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-registry-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.012250 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.012240    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-bound-sa-token\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.012340 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.012329    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a-installation-pull-secrets\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:29:21.249591 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.249549    2563 generic.go:356] "Generic (PLEG): container finished" podID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" containerID="ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95" exitCode=0
Jul 17 21:29:21.249767 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.249634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7465848bb8-f277d" event={"ID":"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a","Type":"ContainerDied","Data":"ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95"}
Jul 17 21:29:21.249767 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.249651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7465848bb8-f277d"
Jul 17 21:29:21.249767 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.249717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7465848bb8-f277d" event={"ID":"522f30fd-2eb0-47e9-a6c1-0ee6e31a893a","Type":"ContainerDied","Data":"77fadf2102a2b81078960fe0a9987a9cd49cc8a8d75680a7f7402a8e7314aaa4"}
Jul 17 21:29:21.249767 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.249738    2563 scope.go:117] "RemoveContainer" containerID="ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95"
Jul 17 21:29:21.258521 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.258499    2563 scope.go:117] "RemoveContainer" containerID="ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95"
Jul 17 21:29:21.258821 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:29:21.258802    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95\": container with ID starting with ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95 not found: ID does not exist" containerID="ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95"
Jul 17 21:29:21.258893 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.258834    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95"} err="failed to get container status \"ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95\": rpc error: code = NotFound desc = could not find container \"ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95\": container with ID starting with ef0dbdbfd3148666a8cc940e77aef4121d439614cf9a871bd1e0a37f8255fc95 not found: ID does not exist"
Jul 17 21:29:21.271029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.270976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7465848bb8-f277d"]
Jul 17 21:29:21.275487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:21.275466    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7465848bb8-f277d"]
Jul 17 21:29:22.601928 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:29:22.601894    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" path="/var/lib/kubelet/pods/522f30fd-2eb0-47e9-a6c1-0ee6e31a893a/volumes"
Jul 17 21:32:16.508713 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:16.508675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:32:16.515986 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:16.515932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:32:16.524589 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:16.524565    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 21:32:34.532291 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532256    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"]
Jul 17 21:32:34.534820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" containerName="registry"
Jul 17 21:32:34.534820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" containerName="registry"
Jul 17 21:32:34.534820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532730    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3371b1-d37d-44cd-8945-e1c7a022306a" containerName="console"
Jul 17 21:32:34.534820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532735    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3371b1-d37d-44cd-8945-e1c7a022306a" containerName="console"
Jul 17 21:32:34.534820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="522f30fd-2eb0-47e9-a6c1-0ee6e31a893a" containerName="registry"
Jul 17 21:32:34.534820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.532805    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3371b1-d37d-44cd-8945-e1c7a022306a" containerName="console"
Jul 17 21:32:34.535725 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.535708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.539399 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.539376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9ws4k\""
Jul 17 21:32:34.539399 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.539397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 21:32:34.539617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.539422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 21:32:34.539774 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.539756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 21:32:34.540078 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.540060    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 21:32:34.584329 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.584295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64e8a846-c582-4d3d-966c-4ae48195e2cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.584474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.584340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e8a846-c582-4d3d-966c-4ae48195e2cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.584474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.584375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8m4\" (UniqueName: \"kubernetes.io/projected/64e8a846-c582-4d3d-966c-4ae48195e2cc-kube-api-access-nc8m4\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.584544 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.584469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64e8a846-c582-4d3d-966c-4ae48195e2cc-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.584544 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.584508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e8a846-c582-4d3d-966c-4ae48195e2cc-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.611820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.611795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"]
Jul 17 21:32:34.685932 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.685894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64e8a846-c582-4d3d-966c-4ae48195e2cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.685932 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.685933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e8a846-c582-4d3d-966c-4ae48195e2cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.686136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.686116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8m4\" (UniqueName: \"kubernetes.io/projected/64e8a846-c582-4d3d-966c-4ae48195e2cc-kube-api-access-nc8m4\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.686198 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.686183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64e8a846-c582-4d3d-966c-4ae48195e2cc-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.686258 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.686216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e8a846-c582-4d3d-966c-4ae48195e2cc-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.686575 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.686550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64e8a846-c582-4d3d-966c-4ae48195e2cc-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.688359 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.688329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e8a846-c582-4d3d-966c-4ae48195e2cc-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.688487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.688439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64e8a846-c582-4d3d-966c-4ae48195e2cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.688546 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.688513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e8a846-c582-4d3d-966c-4ae48195e2cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.696878 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.696856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8m4\" (UniqueName: \"kubernetes.io/projected/64e8a846-c582-4d3d-966c-4ae48195e2cc-kube-api-access-nc8m4\") pod \"opendatahub-operator-controller-manager-964b768d-lnxjh\" (UID: \"64e8a846-c582-4d3d-966c-4ae48195e2cc\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.846891 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.846809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:34.988432 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.988305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"]
Jul 17 21:32:34.991462 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:32:34.991429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e8a846_c582_4d3d_966c_4ae48195e2cc.slice/crio-756ede30c6f2b4d3f11ad979409e9f45a0684bdf7a3da9d21a5cf7456b440737 WatchSource:0}: Error finding container 756ede30c6f2b4d3f11ad979409e9f45a0684bdf7a3da9d21a5cf7456b440737: Status 404 returned error can't find the container with id 756ede30c6f2b4d3f11ad979409e9f45a0684bdf7a3da9d21a5cf7456b440737
Jul 17 21:32:34.993153 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:34.993135    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:32:35.993962 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:35.993916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh" event={"ID":"64e8a846-c582-4d3d-966c-4ae48195e2cc","Type":"ContainerStarted","Data":"756ede30c6f2b4d3f11ad979409e9f45a0684bdf7a3da9d21a5cf7456b440737"}
Jul 17 21:32:39.008031 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:39.007989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh" event={"ID":"64e8a846-c582-4d3d-966c-4ae48195e2cc","Type":"ContainerStarted","Data":"b409abf3d17189cd29ef61bc0d59aaf5dd083c131713deda0a333932c723d063"}
Jul 17 21:32:40.013684 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:40.013643    2563 generic.go:356] "Generic (PLEG): container finished" podID="64e8a846-c582-4d3d-966c-4ae48195e2cc" containerID="b409abf3d17189cd29ef61bc0d59aaf5dd083c131713deda0a333932c723d063" exitCode=0
Jul 17 21:32:40.014067 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:40.013735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh" event={"ID":"64e8a846-c582-4d3d-966c-4ae48195e2cc","Type":"ContainerDied","Data":"b409abf3d17189cd29ef61bc0d59aaf5dd083c131713deda0a333932c723d063"}
Jul 17 21:32:41.020174 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:41.020138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh" event={"ID":"64e8a846-c582-4d3d-966c-4ae48195e2cc","Type":"ContainerStarted","Data":"04b53e81ea41a906e940eb8f2f52a27f9b50bf97986694a18d370a041c3cde46"}
Jul 17 21:32:41.020591 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:41.020254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:32:41.042244 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:41.042195    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh" podStartSLOduration=2.60652801 podStartE2EDuration="7.042177682s" totalImagesPullingTime="3.416477741s" totalInitContainerRuntime="1.019171931s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:41.039630329 +0000 UTC m=+324.951870608" watchObservedRunningTime="2026-07-17 21:32:41.042177682 +0000 UTC m=+324.954417969"
Jul 17 21:32:52.026930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:32:52.026900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-lnxjh"
Jul 17 21:33:04.084277 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.084240    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"]
Jul 17 21:33:04.093685 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.093660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.098906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.098882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 21:33:04.099797 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.099776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 21:33:04.099927 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.099831    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:33:04.099927 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.099889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5q5j2\""
Jul 17 21:33:04.100064 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.099955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 21:33:04.100204 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.100186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 21:33:04.101197 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.101177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"]
Jul 17 21:33:04.231922 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.231889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghjm\" (UniqueName: \"kubernetes.io/projected/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-kube-api-access-tghjm\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.232113 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.231943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-manager-config\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.232113 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.231973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-cert\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.232113 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.232058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-metrics-cert\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.332742 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.332706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghjm\" (UniqueName: \"kubernetes.io/projected/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-kube-api-access-tghjm\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.332942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.332769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-manager-config\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.332942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.332813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-cert\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.332942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.332854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-metrics-cert\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.334229 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.334203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-manager-config\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.335927 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.335863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-cert\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.336227 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.336206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-metrics-cert\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.342774 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.342735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghjm\" (UniqueName: \"kubernetes.io/projected/31a4d9f3-81cd-4c53-a978-82ea947e0f5f-kube-api-access-tghjm\") pod \"lws-controller-manager-7bbc9cc6cd-lkmtb\" (UID: \"31a4d9f3-81cd-4c53-a978-82ea947e0f5f\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.403724 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.403683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:04.539642 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:04.539617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"]
Jul 17 21:33:04.542386 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:33:04.542358    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a4d9f3_81cd_4c53_a978_82ea947e0f5f.slice/crio-abd76a4d95deffc9a276e62d6e2c1d7bd3198203a836c0c5e0f914aa26c42a88 WatchSource:0}: Error finding container abd76a4d95deffc9a276e62d6e2c1d7bd3198203a836c0c5e0f914aa26c42a88: Status 404 returned error can't find the container with id abd76a4d95deffc9a276e62d6e2c1d7bd3198203a836c0c5e0f914aa26c42a88
Jul 17 21:33:05.129441 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:05.129409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb" event={"ID":"31a4d9f3-81cd-4c53-a978-82ea947e0f5f","Type":"ContainerStarted","Data":"abd76a4d95deffc9a276e62d6e2c1d7bd3198203a836c0c5e0f914aa26c42a88"}
Jul 17 21:33:05.951307 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:05.951275    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"]
Jul 17 21:33:05.955040 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:05.955017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:05.957826 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:05.957801    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 17 21:33:05.957974 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:05.957954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pw745\""
Jul 17 21:33:05.967365 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:05.967105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"]
Jul 17 21:33:06.047410 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.047200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9524360e-21e3-4701-bb03-441a60607603-cert\") pod \"kserve-controller-manager-d6f4b65b8-lkgss\" (UID: \"9524360e-21e3-4701-bb03-441a60607603\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:06.047410 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.047273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmp2\" (UniqueName: \"kubernetes.io/projected/9524360e-21e3-4701-bb03-441a60607603-kube-api-access-vbmp2\") pod \"kserve-controller-manager-d6f4b65b8-lkgss\" (UID: \"9524360e-21e3-4701-bb03-441a60607603\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:06.148448 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.148411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9524360e-21e3-4701-bb03-441a60607603-cert\") pod \"kserve-controller-manager-d6f4b65b8-lkgss\" (UID: \"9524360e-21e3-4701-bb03-441a60607603\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:06.148944 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.148471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmp2\" (UniqueName: \"kubernetes.io/projected/9524360e-21e3-4701-bb03-441a60607603-kube-api-access-vbmp2\") pod \"kserve-controller-manager-d6f4b65b8-lkgss\" (UID: \"9524360e-21e3-4701-bb03-441a60607603\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:06.151553 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.151530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9524360e-21e3-4701-bb03-441a60607603-cert\") pod \"kserve-controller-manager-d6f4b65b8-lkgss\" (UID: \"9524360e-21e3-4701-bb03-441a60607603\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:06.162158 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.162135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmp2\" (UniqueName: \"kubernetes.io/projected/9524360e-21e3-4701-bb03-441a60607603-kube-api-access-vbmp2\") pod \"kserve-controller-manager-d6f4b65b8-lkgss\" (UID: \"9524360e-21e3-4701-bb03-441a60607603\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:06.270796 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:06.270713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:07.375952 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:07.375930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"]
Jul 17 21:33:07.378839 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:33:07.378812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9524360e_21e3_4701_bb03_441a60607603.slice/crio-88ef286c6cbc28ed02fd0b339c1698631789fa4388e07565307eebfe1f4ab3ba WatchSource:0}: Error finding container 88ef286c6cbc28ed02fd0b339c1698631789fa4388e07565307eebfe1f4ab3ba: Status 404 returned error can't find the container with id 88ef286c6cbc28ed02fd0b339c1698631789fa4388e07565307eebfe1f4ab3ba
Jul 17 21:33:08.146974 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:08.146888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss" event={"ID":"9524360e-21e3-4701-bb03-441a60607603","Type":"ContainerStarted","Data":"88ef286c6cbc28ed02fd0b339c1698631789fa4388e07565307eebfe1f4ab3ba"}
Jul 17 21:33:08.148560 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:08.148529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb" event={"ID":"31a4d9f3-81cd-4c53-a978-82ea947e0f5f","Type":"ContainerStarted","Data":"ae8f86b57d4667629b55261b3cece264b3ac7d35e717df94132a1ddd7e02995d"}
Jul 17 21:33:08.148736 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:08.148682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:08.166694 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:08.166641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb" podStartSLOduration=1.422896932 podStartE2EDuration="4.166624709s" totalImagesPullingTime="2.743727777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:08.165954842 +0000 UTC m=+352.078195133" watchObservedRunningTime="2026-07-17 21:33:08.166624709 +0000 UTC m=+352.078864989"
Jul 17 21:33:10.158123 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:10.158038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss" event={"ID":"9524360e-21e3-4701-bb03-441a60607603","Type":"ContainerStarted","Data":"8e07fcbbbfbb052945b075615e0df9ed38afb2de8a576566bef8eec4405ff51b"}
Jul 17 21:33:10.158123 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:10.158113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:33:10.175072 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:10.175024    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss" podStartSLOduration=2.881868448 podStartE2EDuration="5.17501011s" totalImagesPullingTime="2.293141662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:10.172903277 +0000 UTC m=+354.085143565" watchObservedRunningTime="2026-07-17 21:33:10.17501011 +0000 UTC m=+354.087250398"
Jul 17 21:33:19.155086 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:19.155053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-lkmtb"
Jul 17 21:33:41.167995 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:33:41.167968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lkgss"
Jul 17 21:34:13.551998 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.551968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c765ccb77-5755m"]
Jul 17 21:34:13.555310 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.555287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.557745 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.557708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 21:34:13.557856 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.557803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 21:34:13.557856 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.557840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 21:34:13.558909 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.558887    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 21:34:13.559018 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.558924    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 21:34:13.559018 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.558896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cnsv4\""
Jul 17 21:34:13.562295 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.562273    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 21:34:13.566740 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.566717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c765ccb77-5755m"]
Jul 17 21:34:13.721175 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-serving-cert\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.721360 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-trusted-ca-bundle\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.721360 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-oauth-config\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.721360 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-config\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.721487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-oauth-serving-cert\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.721487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-service-ca\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.721487 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.721411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25k59\" (UniqueName: \"kubernetes.io/projected/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-kube-api-access-25k59\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822663 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-oauth-serving-cert\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822663 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-service-ca\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822663 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25k59\" (UniqueName: \"kubernetes.io/projected/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-kube-api-access-25k59\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822663 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-serving-cert\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-trusted-ca-bundle\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-oauth-config\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.822956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.822728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-config\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.823412 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.823383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-service-ca\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.823567 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.823551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-trusted-ca-bundle\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.823637 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.823503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-oauth-serving-cert\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.823785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.823765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-config\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.825306 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.825281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-oauth-config\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.825463 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.825449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-console-serving-cert\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.830372 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.830350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25k59\" (UniqueName: \"kubernetes.io/projected/0e0db462-8e7f-4041-bdc7-4037ea7e23c4-kube-api-access-25k59\") pod \"console-c765ccb77-5755m\" (UID: \"0e0db462-8e7f-4041-bdc7-4037ea7e23c4\") " pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:13.867016 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:13.866988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:14.010345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:14.010322    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c765ccb77-5755m"]
Jul 17 21:34:14.012951 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:34:14.012924    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0db462_8e7f_4041_bdc7_4037ea7e23c4.slice/crio-1c5956fee9ac5bb60c38ae9e312462ca1e5d06d1cbbd2f00b7db0a40067e4ccc WatchSource:0}: Error finding container 1c5956fee9ac5bb60c38ae9e312462ca1e5d06d1cbbd2f00b7db0a40067e4ccc: Status 404 returned error can't find the container with id 1c5956fee9ac5bb60c38ae9e312462ca1e5d06d1cbbd2f00b7db0a40067e4ccc
Jul 17 21:34:14.436708 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:14.436674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c765ccb77-5755m" event={"ID":"0e0db462-8e7f-4041-bdc7-4037ea7e23c4","Type":"ContainerStarted","Data":"018e5bf1c32a7be2f6bfe8898adef9343956dec6978f576e8cd13657cb705de9"}
Jul 17 21:34:14.436889 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:14.436720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c765ccb77-5755m" event={"ID":"0e0db462-8e7f-4041-bdc7-4037ea7e23c4","Type":"ContainerStarted","Data":"1c5956fee9ac5bb60c38ae9e312462ca1e5d06d1cbbd2f00b7db0a40067e4ccc"}
Jul 17 21:34:14.455098 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:14.455048    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c765ccb77-5755m" podStartSLOduration=1.455033055 podStartE2EDuration="1.455033055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:14.45358972 +0000 UTC m=+418.365830010" watchObservedRunningTime="2026-07-17 21:34:14.455033055 +0000 UTC m=+418.367273343"
Jul 17 21:34:23.867915 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:23.867872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:23.868317 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:23.867934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:23.872833 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:23.872808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:24.484691 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:24.484655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c765ccb77-5755m"
Jul 17 21:34:27.716023 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.715989    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"]
Jul 17 21:34:27.719682 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.719666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:27.723808 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.723780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 21:34:27.723956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.723899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhf9w\""
Jul 17 21:34:27.724039 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.724022    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 21:34:27.740403 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.740377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"]
Jul 17 21:34:27.746842 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.746814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50786f50-865c-4111-af53-cc2f2d6107b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:27.746978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.746848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgpw\" (UniqueName: \"kubernetes.io/projected/50786f50-865c-4111-af53-cc2f2d6107b7-kube-api-access-9tgpw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:27.847559 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.847509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50786f50-865c-4111-af53-cc2f2d6107b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:27.847559 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.847564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgpw\" (UniqueName: \"kubernetes.io/projected/50786f50-865c-4111-af53-cc2f2d6107b7-kube-api-access-9tgpw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:27.847913 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.847894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50786f50-865c-4111-af53-cc2f2d6107b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:27.858731 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:27.858710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgpw\" (UniqueName: \"kubernetes.io/projected/50786f50-865c-4111-af53-cc2f2d6107b7-kube-api-access-9tgpw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:28.030342 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:28.030239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:28.172757 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:28.172729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"]
Jul 17 21:34:28.174810 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:34:28.174781    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50786f50_865c_4111_af53_cc2f2d6107b7.slice/crio-317203e8b2b1c1a2be8d6e788e5b42a0f0dfcf2bcc0ff9c10b747263e419d44b WatchSource:0}: Error finding container 317203e8b2b1c1a2be8d6e788e5b42a0f0dfcf2bcc0ff9c10b747263e419d44b: Status 404 returned error can't find the container with id 317203e8b2b1c1a2be8d6e788e5b42a0f0dfcf2bcc0ff9c10b747263e419d44b
Jul 17 21:34:28.505749 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:28.505705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh" event={"ID":"50786f50-865c-4111-af53-cc2f2d6107b7","Type":"ContainerStarted","Data":"317203e8b2b1c1a2be8d6e788e5b42a0f0dfcf2bcc0ff9c10b747263e419d44b"}
Jul 17 21:34:34.535468 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:34.535427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh" event={"ID":"50786f50-865c-4111-af53-cc2f2d6107b7","Type":"ContainerStarted","Data":"ab73f1953d943e6d2f882b43099a0f41987288b69a9c04a602f2634c8e929f91"}
Jul 17 21:34:34.535954 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:34.535582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:34.560029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:34.559973    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh" podStartSLOduration=1.7562607350000001 podStartE2EDuration="7.559956958s" totalImagesPullingTime="5.803696223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:34.557223888 +0000 UTC m=+438.469464177" watchObservedRunningTime="2026-07-17 21:34:34.559956958 +0000 UTC m=+438.472197247"
Jul 17 21:34:45.542712 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:45.542626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:47.428430 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.428393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"]
Jul 17 21:34:47.428847 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.428589    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" containerName="manager" containerID="cri-o://ab73f1953d943e6d2f882b43099a0f41987288b69a9c04a602f2634c8e929f91" gracePeriod=2
Jul 17 21:34:47.446970 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.446941    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"]
Jul 17 21:34:47.455428 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:34:47.455388    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"50786f50-865c-4111-af53-cc2f2d6107b7\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\": pods \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:47.491467 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.491433    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"]
Jul 17 21:34:47.492031 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.492004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" containerName="manager"
Jul 17 21:34:47.492031 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.492034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" containerName="manager"
Jul 17 21:34:47.492197 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.492134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" containerName="manager"
Jul 17 21:34:47.495857 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.495839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:34:47.498254 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.498233    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6vzr6\""
Jul 17 21:34:47.511095 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.511073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"]
Jul 17 21:34:47.536880 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:34:47.536848    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:47.599250 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.599221    2563 generic.go:356] "Generic (PLEG): container finished" podID="50786f50-865c-4111-af53-cc2f2d6107b7" containerID="ab73f1953d943e6d2f882b43099a0f41987288b69a9c04a602f2634c8e929f91" exitCode=0
Jul 17 21:34:47.619223 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.619190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xh9\" (UniqueName: \"kubernetes.io/projected/2e01cf0c-c4d9-469e-865c-91fd1f63ed8c-kube-api-access-b2xh9\") pod \"limitador-operator-controller-manager-859d7b78bd-p86js\" (UID: \"2e01cf0c-c4d9-469e-865c-91fd1f63ed8c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:34:47.674787 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.674761    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:47.676947 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:34:47.676922    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hrrqh\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:47.720431 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.720349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xh9\" (UniqueName: \"kubernetes.io/projected/2e01cf0c-c4d9-469e-865c-91fd1f63ed8c-kube-api-access-b2xh9\") pod \"limitador-operator-controller-manager-859d7b78bd-p86js\" (UID: \"2e01cf0c-c4d9-469e-865c-91fd1f63ed8c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:34:47.729222 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.729193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xh9\" (UniqueName: \"kubernetes.io/projected/2e01cf0c-c4d9-469e-865c-91fd1f63ed8c-kube-api-access-b2xh9\") pod \"limitador-operator-controller-manager-859d7b78bd-p86js\" (UID: \"2e01cf0c-c4d9-469e-865c-91fd1f63ed8c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:34:47.818628 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.818564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:34:47.820821 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.820801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgpw\" (UniqueName: \"kubernetes.io/projected/50786f50-865c-4111-af53-cc2f2d6107b7-kube-api-access-9tgpw\") pod \"50786f50-865c-4111-af53-cc2f2d6107b7\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") "
Jul 17 21:34:47.820885 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.820864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50786f50-865c-4111-af53-cc2f2d6107b7-extensions-socket-volume\") pod \"50786f50-865c-4111-af53-cc2f2d6107b7\" (UID: \"50786f50-865c-4111-af53-cc2f2d6107b7\") "
Jul 17 21:34:47.821316 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.821287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50786f50-865c-4111-af53-cc2f2d6107b7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "50786f50-865c-4111-af53-cc2f2d6107b7" (UID: "50786f50-865c-4111-af53-cc2f2d6107b7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:34:47.823648 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.823624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50786f50-865c-4111-af53-cc2f2d6107b7-kube-api-access-9tgpw" (OuterVolumeSpecName: "kube-api-access-9tgpw") pod "50786f50-865c-4111-af53-cc2f2d6107b7" (UID: "50786f50-865c-4111-af53-cc2f2d6107b7"). InnerVolumeSpecName "kube-api-access-9tgpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:34:47.921795 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.921763    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgpw\" (UniqueName: \"kubernetes.io/projected/50786f50-865c-4111-af53-cc2f2d6107b7-kube-api-access-9tgpw\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:34:47.921795 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.921792    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50786f50-865c-4111-af53-cc2f2d6107b7-extensions-socket-volume\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:34:47.977991 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:47.977961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"]
Jul 17 21:34:47.980522 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:34:47.980494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e01cf0c_c4d9_469e_865c_91fd1f63ed8c.slice/crio-335d013e4eda52ef256c7b1c6c184dca6e07f8db77d8ecf322465fe08127c468 WatchSource:0}: Error finding container 335d013e4eda52ef256c7b1c6c184dca6e07f8db77d8ecf322465fe08127c468: Status 404 returned error can't find the container with id 335d013e4eda52ef256c7b1c6c184dca6e07f8db77d8ecf322465fe08127c468
Jul 17 21:34:48.605179 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:48.605136    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50786f50-865c-4111-af53-cc2f2d6107b7" path="/var/lib/kubelet/pods/50786f50-865c-4111-af53-cc2f2d6107b7/volumes"
Jul 17 21:34:48.609776 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:48.609748    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hrrqh"
Jul 17 21:34:48.610532 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:48.609767    2563 scope.go:117] "RemoveContainer" containerID="ab73f1953d943e6d2f882b43099a0f41987288b69a9c04a602f2634c8e929f91"
Jul 17 21:34:48.611548 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:48.611515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js" event={"ID":"2e01cf0c-c4d9-469e-865c-91fd1f63ed8c","Type":"ContainerStarted","Data":"335d013e4eda52ef256c7b1c6c184dca6e07f8db77d8ecf322465fe08127c468"}
Jul 17 21:34:49.620511 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:49.620410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js" event={"ID":"2e01cf0c-c4d9-469e-865c-91fd1f63ed8c","Type":"ContainerStarted","Data":"f54a23bcb8b73aca8e45e66795e2a172bc9aff502726d52bf8410570a3e349be"}
Jul 17 21:34:49.620511 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:49.620477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:34:49.645223 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:34:49.645175    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js" podStartSLOduration=1.3326037450000001 podStartE2EDuration="2.64515912s" totalImagesPullingTime="1.312555375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:49.643302825 +0000 UTC m=+453.555543114" watchObservedRunningTime="2026-07-17 21:34:49.64515912 +0000 UTC m=+453.557399409"
Jul 17 21:35:00.628282 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:00.628250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p86js"
Jul 17 21:35:22.357244 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.357204    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:22.362686 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.362663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.365146 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.365111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kf4g2\""
Jul 17 21:35:22.365240 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.365111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 21:35:22.368981 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.368954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:22.398450 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.398421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9789b94f-ad9e-4ef2-ba99-b70324c50832-config-file\") pod \"limitador-limitador-64c94fd64f-d2dgn\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.398592 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.398459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7g9\" (UniqueName: \"kubernetes.io/projected/9789b94f-ad9e-4ef2-ba99-b70324c50832-kube-api-access-8k7g9\") pod \"limitador-limitador-64c94fd64f-d2dgn\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.458728 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.458692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:22.499616 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.499563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9789b94f-ad9e-4ef2-ba99-b70324c50832-config-file\") pod \"limitador-limitador-64c94fd64f-d2dgn\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.499794 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.499623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7g9\" (UniqueName: \"kubernetes.io/projected/9789b94f-ad9e-4ef2-ba99-b70324c50832-kube-api-access-8k7g9\") pod \"limitador-limitador-64c94fd64f-d2dgn\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.500264 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.500239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9789b94f-ad9e-4ef2-ba99-b70324c50832-config-file\") pod \"limitador-limitador-64c94fd64f-d2dgn\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.508712 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.508684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7g9\" (UniqueName: \"kubernetes.io/projected/9789b94f-ad9e-4ef2-ba99-b70324c50832-kube-api-access-8k7g9\") pod \"limitador-limitador-64c94fd64f-d2dgn\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.673774 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.673742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:22.823036 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:22.822975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:22.825769 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:35:22.825739    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9789b94f_ad9e_4ef2_ba99_b70324c50832.slice/crio-8da8727abfb01b289fdb7658e6d989c5eea421e948ac61bbc93cb13e861d59c8 WatchSource:0}: Error finding container 8da8727abfb01b289fdb7658e6d989c5eea421e948ac61bbc93cb13e861d59c8: Status 404 returned error can't find the container with id 8da8727abfb01b289fdb7658e6d989c5eea421e948ac61bbc93cb13e861d59c8
Jul 17 21:35:23.777751 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:23.777706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn" event={"ID":"9789b94f-ad9e-4ef2-ba99-b70324c50832","Type":"ContainerStarted","Data":"8da8727abfb01b289fdb7658e6d989c5eea421e948ac61bbc93cb13e861d59c8"}
Jul 17 21:35:25.790135 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:25.790104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn" event={"ID":"9789b94f-ad9e-4ef2-ba99-b70324c50832","Type":"ContainerStarted","Data":"26985c13fe2a3ce98a352bd7e93459a6d350fd1c71d988da5ee56b21bed7c388"}
Jul 17 21:35:25.790552 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:25.790168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:25.805920 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:25.805866    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn" podStartSLOduration=0.998632869 podStartE2EDuration="3.805849745s" totalImagesPullingTime="2.807216876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:25.80466632 +0000 UTC m=+489.716906612" watchObservedRunningTime="2026-07-17 21:35:25.805849745 +0000 UTC m=+489.718090052"
Jul 17 21:35:36.796971 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:36.796941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:37.419342 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:37.419302    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:37.419525 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:37.419488    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn" podUID="9789b94f-ad9e-4ef2-ba99-b70324c50832" containerName="limitador" containerID="cri-o://26985c13fe2a3ce98a352bd7e93459a6d350fd1c71d988da5ee56b21bed7c388" gracePeriod=30
Jul 17 21:35:37.845157 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:37.845124    2563 generic.go:356] "Generic (PLEG): container finished" podID="9789b94f-ad9e-4ef2-ba99-b70324c50832" containerID="26985c13fe2a3ce98a352bd7e93459a6d350fd1c71d988da5ee56b21bed7c388" exitCode=0
Jul 17 21:35:37.845546 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:37.845182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn" event={"ID":"9789b94f-ad9e-4ef2-ba99-b70324c50832","Type":"ContainerDied","Data":"26985c13fe2a3ce98a352bd7e93459a6d350fd1c71d988da5ee56b21bed7c388"}
Jul 17 21:35:38.374211 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.374183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:38.445855 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.445824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9789b94f-ad9e-4ef2-ba99-b70324c50832-config-file\") pod \"9789b94f-ad9e-4ef2-ba99-b70324c50832\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") "
Jul 17 21:35:38.446051 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.445976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k7g9\" (UniqueName: \"kubernetes.io/projected/9789b94f-ad9e-4ef2-ba99-b70324c50832-kube-api-access-8k7g9\") pod \"9789b94f-ad9e-4ef2-ba99-b70324c50832\" (UID: \"9789b94f-ad9e-4ef2-ba99-b70324c50832\") "
Jul 17 21:35:38.446214 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.446191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9789b94f-ad9e-4ef2-ba99-b70324c50832-config-file" (OuterVolumeSpecName: "config-file") pod "9789b94f-ad9e-4ef2-ba99-b70324c50832" (UID: "9789b94f-ad9e-4ef2-ba99-b70324c50832"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:35:38.448229 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.448207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9789b94f-ad9e-4ef2-ba99-b70324c50832-kube-api-access-8k7g9" (OuterVolumeSpecName: "kube-api-access-8k7g9") pod "9789b94f-ad9e-4ef2-ba99-b70324c50832" (UID: "9789b94f-ad9e-4ef2-ba99-b70324c50832"). InnerVolumeSpecName "kube-api-access-8k7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:35:38.547011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.546932    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7g9\" (UniqueName: \"kubernetes.io/projected/9789b94f-ad9e-4ef2-ba99-b70324c50832-kube-api-access-8k7g9\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:35:38.547011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.546962    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9789b94f-ad9e-4ef2-ba99-b70324c50832-config-file\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:35:38.851429 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.851336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn" event={"ID":"9789b94f-ad9e-4ef2-ba99-b70324c50832","Type":"ContainerDied","Data":"8da8727abfb01b289fdb7658e6d989c5eea421e948ac61bbc93cb13e861d59c8"}
Jul 17 21:35:38.851429 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.851358    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"
Jul 17 21:35:38.851429 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.851396    2563 scope.go:117] "RemoveContainer" containerID="26985c13fe2a3ce98a352bd7e93459a6d350fd1c71d988da5ee56b21bed7c388"
Jul 17 21:35:38.869413 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.869382    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:38.873609 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.873579    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d2dgn"]
Jul 17 21:35:38.946843 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.946808    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"]
Jul 17 21:35:38.947317 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.947297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9789b94f-ad9e-4ef2-ba99-b70324c50832" containerName="limitador"
Jul 17 21:35:38.947370 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.947324    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9789b94f-ad9e-4ef2-ba99-b70324c50832" containerName="limitador"
Jul 17 21:35:38.947438 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.947426    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9789b94f-ad9e-4ef2-ba99-b70324c50832" containerName="limitador"
Jul 17 21:35:38.952049 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.952023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:38.954474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.954453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 21:35:38.954723 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.954454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:35:38.954850 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.954454    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 21:35:38.954850 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.954487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wm8dh\""
Jul 17 21:35:38.957549 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:38.957526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"]
Jul 17 21:35:39.050143 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.050104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97b8d\" (UniqueName: \"kubernetes.io/projected/e24a9d21-c7ec-446d-82e6-7aca444279e1-kube-api-access-97b8d\") pod \"postgres-69fb4594cf-6vgvh\" (UID: \"e24a9d21-c7ec-446d-82e6-7aca444279e1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.050143 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.050153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e24a9d21-c7ec-446d-82e6-7aca444279e1-data\") pod \"postgres-69fb4594cf-6vgvh\" (UID: \"e24a9d21-c7ec-446d-82e6-7aca444279e1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.151187 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.151093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97b8d\" (UniqueName: \"kubernetes.io/projected/e24a9d21-c7ec-446d-82e6-7aca444279e1-kube-api-access-97b8d\") pod \"postgres-69fb4594cf-6vgvh\" (UID: \"e24a9d21-c7ec-446d-82e6-7aca444279e1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.151187 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.151146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e24a9d21-c7ec-446d-82e6-7aca444279e1-data\") pod \"postgres-69fb4594cf-6vgvh\" (UID: \"e24a9d21-c7ec-446d-82e6-7aca444279e1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.151531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.151515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e24a9d21-c7ec-446d-82e6-7aca444279e1-data\") pod \"postgres-69fb4594cf-6vgvh\" (UID: \"e24a9d21-c7ec-446d-82e6-7aca444279e1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.159455 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.159428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97b8d\" (UniqueName: \"kubernetes.io/projected/e24a9d21-c7ec-446d-82e6-7aca444279e1-kube-api-access-97b8d\") pod \"postgres-69fb4594cf-6vgvh\" (UID: \"e24a9d21-c7ec-446d-82e6-7aca444279e1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.264868 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.264824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:39.401921 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.401893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"]
Jul 17 21:35:39.404430 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:35:39.404399    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24a9d21_c7ec_446d_82e6_7aca444279e1.slice/crio-d309006647703df41c42919c55c51ff5e55f0a9cd8812998b8f15dbe3c7eec83 WatchSource:0}: Error finding container d309006647703df41c42919c55c51ff5e55f0a9cd8812998b8f15dbe3c7eec83: Status 404 returned error can't find the container with id d309006647703df41c42919c55c51ff5e55f0a9cd8812998b8f15dbe3c7eec83
Jul 17 21:35:39.858841 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:39.858801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh" event={"ID":"e24a9d21-c7ec-446d-82e6-7aca444279e1","Type":"ContainerStarted","Data":"d309006647703df41c42919c55c51ff5e55f0a9cd8812998b8f15dbe3c7eec83"}
Jul 17 21:35:40.606617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:40.606560    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9789b94f-ad9e-4ef2-ba99-b70324c50832" path="/var/lib/kubelet/pods/9789b94f-ad9e-4ef2-ba99-b70324c50832/volumes"
Jul 17 21:35:44.887024 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:44.886929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh" event={"ID":"e24a9d21-c7ec-446d-82e6-7aca444279e1","Type":"ContainerStarted","Data":"9a2cea213732c8c434ccd0f532bbd42e342ca7a8884e206ab5d16517c0dafbc3"}
Jul 17 21:35:44.887377 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:44.887034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:35:44.904785 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:44.904729    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh" podStartSLOduration=1.794910487 podStartE2EDuration="6.904713996s" totalImagesPullingTime="5.109803509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:44.902726797 +0000 UTC m=+508.814967086" watchObservedRunningTime="2026-07-17 21:35:44.904713996 +0000 UTC m=+508.816954284"
Jul 17 21:35:50.921711 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:35:50.921677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6vgvh"
Jul 17 21:36:32.637248 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.637213    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"]
Jul 17 21:36:32.639962 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.639944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.642232 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.642213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gw494\""
Jul 17 21:36:32.642901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.642884    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 17 21:36:32.649502 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.649479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"]
Jul 17 21:36:32.660685 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.660663    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-s72bv"]
Jul 17 21:36:32.662958 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.662877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.675535 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.675513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-s72bv"]
Jul 17 21:36:32.715681 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.715651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxtc\" (UniqueName: \"kubernetes.io/projected/aeb09007-48a1-4669-ad10-c05bc2ac93d5-kube-api-access-5fxtc\") pod \"payload-processing-5bf47cd988-s72bv\" (UID: \"aeb09007-48a1-4669-ad10-c05bc2ac93d5\") " pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.715827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.715725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctt2\" (UniqueName: \"kubernetes.io/projected/d4e797a3-5d05-4c8b-ad1d-2dad0f014daf-kube-api-access-sctt2\") pod \"payload-pre-processing-7677b774bb-x5xsr\" (UID: \"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.715827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.715778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb09007-48a1-4669-ad10-c05bc2ac93d5-plugins-config-volume\") pod \"payload-processing-5bf47cd988-s72bv\" (UID: \"aeb09007-48a1-4669-ad10-c05bc2ac93d5\") " pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.715953 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.715921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e797a3-5d05-4c8b-ad1d-2dad0f014daf-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-x5xsr\" (UID: \"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.816559 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.816521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxtc\" (UniqueName: \"kubernetes.io/projected/aeb09007-48a1-4669-ad10-c05bc2ac93d5-kube-api-access-5fxtc\") pod \"payload-processing-5bf47cd988-s72bv\" (UID: \"aeb09007-48a1-4669-ad10-c05bc2ac93d5\") " pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.816559 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.816568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctt2\" (UniqueName: \"kubernetes.io/projected/d4e797a3-5d05-4c8b-ad1d-2dad0f014daf-kube-api-access-sctt2\") pod \"payload-pre-processing-7677b774bb-x5xsr\" (UID: \"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.816828 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.816621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb09007-48a1-4669-ad10-c05bc2ac93d5-plugins-config-volume\") pod \"payload-processing-5bf47cd988-s72bv\" (UID: \"aeb09007-48a1-4669-ad10-c05bc2ac93d5\") " pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.816828 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.816670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e797a3-5d05-4c8b-ad1d-2dad0f014daf-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-x5xsr\" (UID: \"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.817336 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.817316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb09007-48a1-4669-ad10-c05bc2ac93d5-plugins-config-volume\") pod \"payload-processing-5bf47cd988-s72bv\" (UID: \"aeb09007-48a1-4669-ad10-c05bc2ac93d5\") " pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.817384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.817323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e797a3-5d05-4c8b-ad1d-2dad0f014daf-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-x5xsr\" (UID: \"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.825292 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.825259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctt2\" (UniqueName: \"kubernetes.io/projected/d4e797a3-5d05-4c8b-ad1d-2dad0f014daf-kube-api-access-sctt2\") pod \"payload-pre-processing-7677b774bb-x5xsr\" (UID: \"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.825558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.825540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxtc\" (UniqueName: \"kubernetes.io/projected/aeb09007-48a1-4669-ad10-c05bc2ac93d5-kube-api-access-5fxtc\") pod \"payload-processing-5bf47cd988-s72bv\" (UID: \"aeb09007-48a1-4669-ad10-c05bc2ac93d5\") " pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:32.951565 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.951537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:32.973542 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:32.973519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:33.100974 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:33.100950    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"]
Jul 17 21:36:33.103459 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:36:33.103429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e797a3_5d05_4c8b_ad1d_2dad0f014daf.slice/crio-10b6042339943f06596863436ebe912a4adccf03f3eac152407cf1336766bef7 WatchSource:0}: Error finding container 10b6042339943f06596863436ebe912a4adccf03f3eac152407cf1336766bef7: Status 404 returned error can't find the container with id 10b6042339943f06596863436ebe912a4adccf03f3eac152407cf1336766bef7
Jul 17 21:36:33.124454 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:33.124432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-s72bv"]
Jul 17 21:36:33.126366 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:36:33.126344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb09007_48a1_4669_ad10_c05bc2ac93d5.slice/crio-265bb2959848ddde2c4bbd118eb7aa0b60fd1a95cd741e4d742ea288d09068c1 WatchSource:0}: Error finding container 265bb2959848ddde2c4bbd118eb7aa0b60fd1a95cd741e4d742ea288d09068c1: Status 404 returned error can't find the container with id 265bb2959848ddde2c4bbd118eb7aa0b60fd1a95cd741e4d742ea288d09068c1
Jul 17 21:36:34.111421 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:34.111380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv" event={"ID":"aeb09007-48a1-4669-ad10-c05bc2ac93d5","Type":"ContainerStarted","Data":"265bb2959848ddde2c4bbd118eb7aa0b60fd1a95cd741e4d742ea288d09068c1"}
Jul 17 21:36:34.116128 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:34.116085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr" event={"ID":"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf","Type":"ContainerStarted","Data":"10b6042339943f06596863436ebe912a4adccf03f3eac152407cf1336766bef7"}
Jul 17 21:36:36.127664 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:36.127590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv" event={"ID":"aeb09007-48a1-4669-ad10-c05bc2ac93d5","Type":"ContainerStarted","Data":"eccc33279c2dde2bffbcef95b4dd1c7415cff11e5cffed1877712a55009d22a9"}
Jul 17 21:36:36.128135 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:36.127686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:36.129128 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:36.129105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr" event={"ID":"d4e797a3-5d05-4c8b-ad1d-2dad0f014daf","Type":"ContainerStarted","Data":"bb17f822b5435efabf9a78b4d048168383d378f3d850eb08495589e796c0f4c6"}
Jul 17 21:36:36.129232 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:36.129206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:36.144989 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:36.144943    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv" podStartSLOduration=1.9119217480000001 podStartE2EDuration="4.144928543s" totalImagesPullingTime="2.233006795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:36.143056093 +0000 UTC m=+560.055296382" watchObservedRunningTime="2026-07-17 21:36:36.144928543 +0000 UTC m=+560.057168831"
Jul 17 21:36:36.160150 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:36.160090    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr" podStartSLOduration=1.900906846 podStartE2EDuration="4.16007318s" totalImagesPullingTime="2.259166334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:36.157481876 +0000 UTC m=+560.069722167" watchObservedRunningTime="2026-07-17 21:36:36.16007318 +0000 UTC m=+560.072313471"
Jul 17 21:36:47.135966 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.135922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-s72bv"
Jul 17 21:36:47.136357 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.136002    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-x5xsr"
Jul 17 21:36:47.420824 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.420790    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"]
Jul 17 21:36:47.424310 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.424290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.427058 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.427040    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 21:36:47.427184 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.427108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ghszk\""
Jul 17 21:36:47.427280 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.427113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 21:36:47.427280 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.427185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 21:36:47.432990 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.432968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"]
Jul 17 21:36:47.548752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.548723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a51e4f-ec36-4607-af46-a9059d9928ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.548935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.548765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.548935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.548799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxsq7\" (UniqueName: \"kubernetes.io/projected/13a51e4f-ec36-4607-af46-a9059d9928ab-kube-api-access-mxsq7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.548935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.548839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.548935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.548863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.548935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.548891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.649739 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.649703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxsq7\" (UniqueName: \"kubernetes.io/projected/13a51e4f-ec36-4607-af46-a9059d9928ab-kube-api-access-mxsq7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.649901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.649745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.649901 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.649789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.649976 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.649938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.650029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.650007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a51e4f-ec36-4607-af46-a9059d9928ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.650083 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.650063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.650217 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.650196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.650299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.650281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.650394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.650374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.652281 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.652262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a51e4f-ec36-4607-af46-a9059d9928ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.652580 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.652565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a51e4f-ec36-4607-af46-a9059d9928ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.657248 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.657226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxsq7\" (UniqueName: \"kubernetes.io/projected/13a51e4f-ec36-4607-af46-a9059d9928ab-kube-api-access-mxsq7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx\" (UID: \"13a51e4f-ec36-4607-af46-a9059d9928ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.735176 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.735097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:47.888694 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:47.888657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"]
Jul 17 21:36:47.896836 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:36:47.896807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a51e4f_ec36_4607_af46_a9059d9928ab.slice/crio-cae75917b71a30ee0e75234695981b6a998e378a70bf57f0731ab57efa7cc935 WatchSource:0}: Error finding container cae75917b71a30ee0e75234695981b6a998e378a70bf57f0731ab57efa7cc935: Status 404 returned error can't find the container with id cae75917b71a30ee0e75234695981b6a998e378a70bf57f0731ab57efa7cc935
Jul 17 21:36:48.184446 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:48.184406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx" event={"ID":"13a51e4f-ec36-4607-af46-a9059d9928ab","Type":"ContainerStarted","Data":"cae75917b71a30ee0e75234695981b6a998e378a70bf57f0731ab57efa7cc935"}
Jul 17 21:36:49.190976 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:49.190939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx" event={"ID":"13a51e4f-ec36-4607-af46-a9059d9928ab","Type":"ContainerStarted","Data":"f9846131e07a584816dc5fdff632325ff592f43339bbf53a6a69de546dc6e9cf"}
Jul 17 21:36:49.208681 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:49.208628    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx" podStartSLOduration=1.050038991 podStartE2EDuration="2.208612783s" totalImagesPullingTime="1.158573792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:49.207087585 +0000 UTC m=+573.119327895" watchObservedRunningTime="2026-07-17 21:36:49.208612783 +0000 UTC m=+573.120853063"
Jul 17 21:36:57.736111 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:57.736065    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:57.736631 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:57.736140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:57.748687 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:57.748654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:36:58.245272 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:36:58.245242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx"
Jul 17 21:37:10.203536 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.203495    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"]
Jul 17 21:37:10.206840 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.206817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.209495 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.209474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 17 21:37:10.228861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.228830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.229016 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.228870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.229016 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.228919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.229016 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.228937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c116926b-b349-4a5e-a311-71a0ff5e27d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.229016 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.228958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkktq\" (UniqueName: \"kubernetes.io/projected/c116926b-b349-4a5e-a311-71a0ff5e27d2-kube-api-access-wkktq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.229016 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.228978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.233372 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.233344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"]
Jul 17 21:37:10.329902 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.329865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkktq\" (UniqueName: \"kubernetes.io/projected/c116926b-b349-4a5e-a311-71a0ff5e27d2-kube-api-access-wkktq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.329902 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.329908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330119 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.329959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330119 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.329985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330119 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.330030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330261 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.330201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c116926b-b349-4a5e-a311-71a0ff5e27d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330453 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.330436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330526 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.330500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.330610 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.330574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.332651 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.332625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c116926b-b349-4a5e-a311-71a0ff5e27d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.332847 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.332833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c116926b-b349-4a5e-a311-71a0ff5e27d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.337653 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.337628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkktq\" (UniqueName: \"kubernetes.io/projected/c116926b-b349-4a5e-a311-71a0ff5e27d2-kube-api-access-wkktq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5fv2x\" (UID: \"c116926b-b349-4a5e-a311-71a0ff5e27d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.517492 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.517407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:10.661580 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:10.661553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"]
Jul 17 21:37:10.664079 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:37:10.664046    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc116926b_b349_4a5e_a311_71a0ff5e27d2.slice/crio-a9eaec8d1bfdfc802c28eca1888aa052b952fc8db35000addf836a0f07ceaadc WatchSource:0}: Error finding container a9eaec8d1bfdfc802c28eca1888aa052b952fc8db35000addf836a0f07ceaadc: Status 404 returned error can't find the container with id a9eaec8d1bfdfc802c28eca1888aa052b952fc8db35000addf836a0f07ceaadc
Jul 17 21:37:11.298885 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:11.298855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x" event={"ID":"c116926b-b349-4a5e-a311-71a0ff5e27d2","Type":"ContainerStarted","Data":"9a01a95241d27671bf8b10295393bedfd6480d8b544af71329a209153e5db472"}
Jul 17 21:37:11.299266 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:11.298903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x" event={"ID":"c116926b-b349-4a5e-a311-71a0ff5e27d2","Type":"ContainerStarted","Data":"a9eaec8d1bfdfc802c28eca1888aa052b952fc8db35000addf836a0f07ceaadc"}
Jul 17 21:37:11.318228 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:11.318174    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x" podStartSLOduration=1.153630892 podStartE2EDuration="1.318158542s" totalImagesPullingTime="164.52765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:37:11.316712653 +0000 UTC m=+595.228952941" watchObservedRunningTime="2026-07-17 21:37:11.318158542 +0000 UTC m=+595.230398831"
Jul 17 21:37:16.543564 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:16.543537    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:37:16.546068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:16.546045    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:37:20.518430 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:20.518389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:20.518870 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:20.518468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:20.531029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:20.531006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:37:21.356032 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:37:21.356004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5fv2x"
Jul 17 21:42:16.573069 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:42:16.572973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:42:16.573838 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:42:16.573813    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:47:16.604156 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:47:16.604073    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:47:16.606084 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:47:16.606062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:52:16.632094 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:52:16.632057    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:52:16.633699 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:52:16.633681    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:53:29.113472 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.113440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"]
Jul 17 21:53:29.116907 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.116890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:53:29.119125 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.119101    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:53:29.123423 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.123397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"]
Jul 17 21:53:29.192902 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.192866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8dace2-d6c4-4090-869f-c397e568a275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch\" (UID: \"7e8dace2-d6c4-4090-869f-c397e568a275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:53:29.293411 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.293380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8dace2-d6c4-4090-869f-c397e568a275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch\" (UID: \"7e8dace2-d6c4-4090-869f-c397e568a275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:53:29.294114 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.294091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8dace2-d6c4-4090-869f-c397e568a275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch\" (UID: \"7e8dace2-d6c4-4090-869f-c397e568a275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:53:29.428566 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.428527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:53:29.558135 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.558107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"]
Jul 17 21:53:29.561359 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:53:29.561330    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8dace2_d6c4_4090_869f_c397e568a275.slice/crio-ce2de8f26324e9b1a2460633b7239b01ece374e4f63dac592fe5de3bcf9de407 WatchSource:0}: Error finding container ce2de8f26324e9b1a2460633b7239b01ece374e4f63dac592fe5de3bcf9de407: Status 404 returned error can't find the container with id ce2de8f26324e9b1a2460633b7239b01ece374e4f63dac592fe5de3bcf9de407
Jul 17 21:53:29.563332 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.563314    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:53:29.764144 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:29.764053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" event={"ID":"7e8dace2-d6c4-4090-869f-c397e568a275","Type":"ContainerStarted","Data":"ce2de8f26324e9b1a2460633b7239b01ece374e4f63dac592fe5de3bcf9de407"}
Jul 17 21:53:42.824864 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:42.824828    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e8dace2-d6c4-4090-869f-c397e568a275" containerID="39d995fc105941136c9099a1cd061f260af590f564a6d66c42a6f7bb5e4690a6" exitCode=7
Jul 17 21:53:42.825230 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:42.824913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" event={"ID":"7e8dace2-d6c4-4090-869f-c397e568a275","Type":"ContainerDied","Data":"39d995fc105941136c9099a1cd061f260af590f564a6d66c42a6f7bb5e4690a6"}
Jul 17 21:53:42.825230 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:42.825051    2563 scope.go:117] "RemoveContainer" containerID="39d995fc105941136c9099a1cd061f260af590f564a6d66c42a6f7bb5e4690a6"
Jul 17 21:53:43.831493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:43.831411    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e8dace2-d6c4-4090-869f-c397e568a275" containerID="0c8fbf8148ba0c58dd72b5b22067ff6f816e85b31f459be6bdf2abdcbf11103b" exitCode=7
Jul 17 21:53:43.831493 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:43.831472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" event={"ID":"7e8dace2-d6c4-4090-869f-c397e568a275","Type":"ContainerDied","Data":"0c8fbf8148ba0c58dd72b5b22067ff6f816e85b31f459be6bdf2abdcbf11103b"}
Jul 17 21:53:43.831930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:43.831518    2563 scope.go:117] "RemoveContainer" containerID="39d995fc105941136c9099a1cd061f260af590f564a6d66c42a6f7bb5e4690a6"
Jul 17 21:53:43.831930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:43.831719    2563 scope.go:117] "RemoveContainer" containerID="0c8fbf8148ba0c58dd72b5b22067ff6f816e85b31f459be6bdf2abdcbf11103b"
Jul 17 21:53:43.831997 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:53:43.831948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch_odh-ai-gateway-infra(7e8dace2-d6c4-4090-869f-c397e568a275)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" podUID="7e8dace2-d6c4-4090-869f-c397e568a275"
Jul 17 21:53:44.837959 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:44.837929    2563 scope.go:117] "RemoveContainer" containerID="0c8fbf8148ba0c58dd72b5b22067ff6f816e85b31f459be6bdf2abdcbf11103b"
Jul 17 21:53:44.838361 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:53:44.838110    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch_odh-ai-gateway-infra(7e8dace2-d6c4-4090-869f-c397e568a275)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" podUID="7e8dace2-d6c4-4090-869f-c397e568a275"
Jul 17 21:53:59.597403 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:59.597369    2563 scope.go:117] "RemoveContainer" containerID="0c8fbf8148ba0c58dd72b5b22067ff6f816e85b31f459be6bdf2abdcbf11103b"
Jul 17 21:53:59.908026 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:59.907992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" event={"ID":"7e8dace2-d6c4-4090-869f-c397e568a275","Type":"ContainerStarted","Data":"bf1b7fccf27b81e662a1b6a59bacf8881e5b9d7610a3137b0a0485f36f040372"}
Jul 17 21:53:59.922921 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:53:59.922870    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" podStartSLOduration=18.405712813 podStartE2EDuration="30.922855017s" totalImagesPullingTime="12.517142204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:59.921764278 +0000 UTC m=+1603.834004568" watchObservedRunningTime="2026-07-17 21:53:59.922855017 +0000 UTC m=+1603.835095305"
Jul 17 21:54:00.916352 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:00.916317    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e8dace2-d6c4-4090-869f-c397e568a275" containerID="bf1b7fccf27b81e662a1b6a59bacf8881e5b9d7610a3137b0a0485f36f040372" exitCode=7
Jul 17 21:54:00.916765 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:00.916387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" event={"ID":"7e8dace2-d6c4-4090-869f-c397e568a275","Type":"ContainerDied","Data":"bf1b7fccf27b81e662a1b6a59bacf8881e5b9d7610a3137b0a0485f36f040372"}
Jul 17 21:54:00.916765 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:00.916471    2563 scope.go:117] "RemoveContainer" containerID="0c8fbf8148ba0c58dd72b5b22067ff6f816e85b31f459be6bdf2abdcbf11103b"
Jul 17 21:54:00.916765 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:00.916753    2563 scope.go:117] "RemoveContainer" containerID="bf1b7fccf27b81e662a1b6a59bacf8881e5b9d7610a3137b0a0485f36f040372"
Jul 17 21:54:00.917018 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:54:00.916996    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch_odh-ai-gateway-infra(7e8dace2-d6c4-4090-869f-c397e568a275)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" podUID="7e8dace2-d6c4-4090-869f-c397e568a275"
Jul 17 21:54:00.931509 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:00.931486    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"]
Jul 17 21:54:02.065795 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.065774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:54:02.079500 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.079473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8dace2-d6c4-4090-869f-c397e568a275-maas-api-service-ca\") pod \"7e8dace2-d6c4-4090-869f-c397e568a275\" (UID: \"7e8dace2-d6c4-4090-869f-c397e568a275\") "
Jul 17 21:54:02.079935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.079908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8dace2-d6c4-4090-869f-c397e568a275-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e8dace2-d6c4-4090-869f-c397e568a275" (UID: "7e8dace2-d6c4-4090-869f-c397e568a275"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:02.181362 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.181281    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8dace2-d6c4-4090-869f-c397e568a275-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:54:02.929659 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.929622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch" event={"ID":"7e8dace2-d6c4-4090-869f-c397e568a275","Type":"ContainerDied","Data":"ce2de8f26324e9b1a2460633b7239b01ece374e4f63dac592fe5de3bcf9de407"}
Jul 17 21:54:02.929659 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.929646    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"
Jul 17 21:54:02.929853 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.929697    2563 scope.go:117] "RemoveContainer" containerID="bf1b7fccf27b81e662a1b6a59bacf8881e5b9d7610a3137b0a0485f36f040372"
Jul 17 21:54:02.945112 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.945087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"]
Jul 17 21:54:02.948432 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:02.948408    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-wvpch"]
Jul 17 21:54:04.604258 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:04.604226    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" path="/var/lib/kubelet/pods/7e8dace2-d6c4-4090-869f-c397e568a275/volumes"
Jul 17 21:54:39.167714 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.167679    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"]
Jul 17 21:54:39.168136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.168040    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:54:39.168136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.168050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:54:39.168136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.168077    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:54:39.168136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.168083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:54:39.168136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.168138    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:54:39.168303 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.168146    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:54:39.172295 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.172279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:54:39.174763 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.174745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:54:39.178033 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.178012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"]
Jul 17 21:54:39.323635 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.323568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6529a-7ea2-4d97-86d9-9680ec763f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2\" (UID: \"07d6529a-7ea2-4d97-86d9-9680ec763f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:54:39.424660 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.424572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6529a-7ea2-4d97-86d9-9680ec763f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2\" (UID: \"07d6529a-7ea2-4d97-86d9-9680ec763f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:54:39.425186 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.425164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6529a-7ea2-4d97-86d9-9680ec763f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2\" (UID: \"07d6529a-7ea2-4d97-86d9-9680ec763f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:54:39.483904 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.483870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:54:39.620118 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:39.620088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"]
Jul 17 21:54:39.621987 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:54:39.621961    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d6529a_7ea2_4d97_86d9_9680ec763f6c.slice/crio-f0b2aee9186d5eae5178af48840c3d583d9e5a9a349924182ce4df640106af95 WatchSource:0}: Error finding container f0b2aee9186d5eae5178af48840c3d583d9e5a9a349924182ce4df640106af95: Status 404 returned error can't find the container with id f0b2aee9186d5eae5178af48840c3d583d9e5a9a349924182ce4df640106af95
Jul 17 21:54:40.113221 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:40.113186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerStarted","Data":"46b2d760ae9d68a49d31b9aecd2fca4b7097d34d22ba20940a660bdb277a1fd1"}
Jul 17 21:54:40.113394 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:40.113241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerStarted","Data":"f0b2aee9186d5eae5178af48840c3d583d9e5a9a349924182ce4df640106af95"}
Jul 17 21:54:40.129291 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:40.129202    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" podStartSLOduration=1.1291862799999999 podStartE2EDuration="1.12918628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:40.127577323 +0000 UTC m=+1644.039817614" watchObservedRunningTime="2026-07-17 21:54:40.12918628 +0000 UTC m=+1644.041426614"
Jul 17 21:54:41.119613 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:41.119564    2563 generic.go:356] "Generic (PLEG): container finished" podID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerID="46b2d760ae9d68a49d31b9aecd2fca4b7097d34d22ba20940a660bdb277a1fd1" exitCode=7
Jul 17 21:54:41.120011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:41.119656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerDied","Data":"46b2d760ae9d68a49d31b9aecd2fca4b7097d34d22ba20940a660bdb277a1fd1"}
Jul 17 21:54:41.120011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:41.119854    2563 scope.go:117] "RemoveContainer" containerID="46b2d760ae9d68a49d31b9aecd2fca4b7097d34d22ba20940a660bdb277a1fd1"
Jul 17 21:54:42.126196 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:42.126113    2563 generic.go:356] "Generic (PLEG): container finished" podID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerID="6ba62012bb1564ee7c582118d365219c37b913b20731af31c270bd8cf3f4efac" exitCode=7
Jul 17 21:54:42.126629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:42.126186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerDied","Data":"6ba62012bb1564ee7c582118d365219c37b913b20731af31c270bd8cf3f4efac"}
Jul 17 21:54:42.126629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:42.126282    2563 scope.go:117] "RemoveContainer" containerID="46b2d760ae9d68a49d31b9aecd2fca4b7097d34d22ba20940a660bdb277a1fd1"
Jul 17 21:54:42.126629 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:42.126443    2563 scope.go:117] "RemoveContainer" containerID="6ba62012bb1564ee7c582118d365219c37b913b20731af31c270bd8cf3f4efac"
Jul 17 21:54:42.126769 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:54:42.126672    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2_odh-ai-gateway-infra(07d6529a-7ea2-4d97-86d9-9680ec763f6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c"
Jul 17 21:54:43.133109 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:43.133083    2563 scope.go:117] "RemoveContainer" containerID="6ba62012bb1564ee7c582118d365219c37b913b20731af31c270bd8cf3f4efac"
Jul 17 21:54:43.133520 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:54:43.133259    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2_odh-ai-gateway-infra(07d6529a-7ea2-4d97-86d9-9680ec763f6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c"
Jul 17 21:54:56.599421 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:56.599385    2563 scope.go:117] "RemoveContainer" containerID="6ba62012bb1564ee7c582118d365219c37b913b20731af31c270bd8cf3f4efac"
Jul 17 21:54:57.198304 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:57.198273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerStarted","Data":"4b5bb67fa7a9bc1e4b1c6c6e17e79d3935ec719ec234853805a674e721298ec4"}
Jul 17 21:54:58.205074 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:58.205036    2563 generic.go:356] "Generic (PLEG): container finished" podID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerID="4b5bb67fa7a9bc1e4b1c6c6e17e79d3935ec719ec234853805a674e721298ec4" exitCode=7
Jul 17 21:54:58.205526 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:58.205099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerDied","Data":"4b5bb67fa7a9bc1e4b1c6c6e17e79d3935ec719ec234853805a674e721298ec4"}
Jul 17 21:54:58.205526 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:58.205149    2563 scope.go:117] "RemoveContainer" containerID="6ba62012bb1564ee7c582118d365219c37b913b20731af31c270bd8cf3f4efac"
Jul 17 21:54:58.205526 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:58.205469    2563 scope.go:117] "RemoveContainer" containerID="4b5bb67fa7a9bc1e4b1c6c6e17e79d3935ec719ec234853805a674e721298ec4"
Jul 17 21:54:58.205715 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:54:58.205691    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2_odh-ai-gateway-infra(07d6529a-7ea2-4d97-86d9-9680ec763f6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c"
Jul 17 21:54:58.229011 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:58.228981    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"]
Jul 17 21:54:59.366870 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:59.366845    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:54:59.467150 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:59.467055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6529a-7ea2-4d97-86d9-9680ec763f6c-maas-api-service-ca\") pod \"07d6529a-7ea2-4d97-86d9-9680ec763f6c\" (UID: \"07d6529a-7ea2-4d97-86d9-9680ec763f6c\") "
Jul 17 21:54:59.467421 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:59.467398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d6529a-7ea2-4d97-86d9-9680ec763f6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07d6529a-7ea2-4d97-86d9-9680ec763f6c" (UID: "07d6529a-7ea2-4d97-86d9-9680ec763f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:59.567683 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:54:59.567653    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d6529a-7ea2-4d97-86d9-9680ec763f6c-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:55:00.220275 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:00.220240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2" event={"ID":"07d6529a-7ea2-4d97-86d9-9680ec763f6c","Type":"ContainerDied","Data":"f0b2aee9186d5eae5178af48840c3d583d9e5a9a349924182ce4df640106af95"}
Jul 17 21:55:00.220275 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:00.220270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"
Jul 17 21:55:00.220495 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:00.220303    2563 scope.go:117] "RemoveContainer" containerID="4b5bb67fa7a9bc1e4b1c6c6e17e79d3935ec719ec234853805a674e721298ec4"
Jul 17 21:55:00.242986 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:00.242961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"]
Jul 17 21:55:00.248217 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:00.248181    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-p5kf2"]
Jul 17 21:55:00.602813 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:00.602733    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" path="/var/lib/kubelet/pods/07d6529a-7ea2-4d97-86d9-9680ec763f6c/volumes"
Jul 17 21:55:39.212363 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212284    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"]
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212644    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212695    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212755    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212762    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212768    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:55:39.212788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.212776    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8dace2-d6c4-4090-869f-c397e568a275" containerName="revoke-keys"
Jul 17 21:55:39.215760 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.215744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:39.217993 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.217975    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:55:39.224656 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.224633    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"]
Jul 17 21:55:39.323283 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.323242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6a3bf8-c928-4bc0-93f2-201e302605fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx\" (UID: \"3b6a3bf8-c928-4bc0-93f2-201e302605fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:39.424171 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.424131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6a3bf8-c928-4bc0-93f2-201e302605fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx\" (UID: \"3b6a3bf8-c928-4bc0-93f2-201e302605fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:39.424846 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.424829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6a3bf8-c928-4bc0-93f2-201e302605fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx\" (UID: \"3b6a3bf8-c928-4bc0-93f2-201e302605fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:39.526438 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.526347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:39.668031 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:39.668005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"]
Jul 17 21:55:39.669991 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:55:39.669960    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6a3bf8_c928_4bc0_93f2_201e302605fd.slice/crio-a1be577e4fc92ac852302d52ddd27794e7b68ddec44408d9c95ce3b669f575de WatchSource:0}: Error finding container a1be577e4fc92ac852302d52ddd27794e7b68ddec44408d9c95ce3b669f575de: Status 404 returned error can't find the container with id a1be577e4fc92ac852302d52ddd27794e7b68ddec44408d9c95ce3b669f575de
Jul 17 21:55:40.412390 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:40.412352    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerID="ea5adb5abf8059bd3eca163c34d930816d623fc499864a5aa036a4675e47e538" exitCode=7
Jul 17 21:55:40.412782 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:40.412416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" event={"ID":"3b6a3bf8-c928-4bc0-93f2-201e302605fd","Type":"ContainerDied","Data":"ea5adb5abf8059bd3eca163c34d930816d623fc499864a5aa036a4675e47e538"}
Jul 17 21:55:40.412782 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:40.412469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" event={"ID":"3b6a3bf8-c928-4bc0-93f2-201e302605fd","Type":"ContainerStarted","Data":"a1be577e4fc92ac852302d52ddd27794e7b68ddec44408d9c95ce3b669f575de"}
Jul 17 21:55:40.412782 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:40.412721    2563 scope.go:117] "RemoveContainer" containerID="ea5adb5abf8059bd3eca163c34d930816d623fc499864a5aa036a4675e47e538"
Jul 17 21:55:41.419098 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:41.419070    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerID="1e269cacb3125c13a904a6c34d1c457a7e40e681526beaf046e03964e0236096" exitCode=7
Jul 17 21:55:41.419531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:41.419159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" event={"ID":"3b6a3bf8-c928-4bc0-93f2-201e302605fd","Type":"ContainerDied","Data":"1e269cacb3125c13a904a6c34d1c457a7e40e681526beaf046e03964e0236096"}
Jul 17 21:55:41.419531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:41.419228    2563 scope.go:117] "RemoveContainer" containerID="ea5adb5abf8059bd3eca163c34d930816d623fc499864a5aa036a4675e47e538"
Jul 17 21:55:41.419531 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:41.419324    2563 scope.go:117] "RemoveContainer" containerID="1e269cacb3125c13a904a6c34d1c457a7e40e681526beaf046e03964e0236096"
Jul 17 21:55:41.419531 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:55:41.419517    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx_odh-ai-gateway-infra(3b6a3bf8-c928-4bc0-93f2-201e302605fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd"
Jul 17 21:55:42.425904 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:42.425876    2563 scope.go:117] "RemoveContainer" containerID="1e269cacb3125c13a904a6c34d1c457a7e40e681526beaf046e03964e0236096"
Jul 17 21:55:42.426287 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:55:42.426058    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx_odh-ai-gateway-infra(3b6a3bf8-c928-4bc0-93f2-201e302605fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd"
Jul 17 21:55:55.596897 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:55.596866    2563 scope.go:117] "RemoveContainer" containerID="1e269cacb3125c13a904a6c34d1c457a7e40e681526beaf046e03964e0236096"
Jul 17 21:55:56.491911 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:56.491868    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerID="ee90260e9de8159e710eb2679205c72790b2098d64fe5cba9e30a801137389e4" exitCode=7
Jul 17 21:55:56.492089 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:56.491942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" event={"ID":"3b6a3bf8-c928-4bc0-93f2-201e302605fd","Type":"ContainerDied","Data":"ee90260e9de8159e710eb2679205c72790b2098d64fe5cba9e30a801137389e4"}
Jul 17 21:55:56.492089 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:56.491996    2563 scope.go:117] "RemoveContainer" containerID="1e269cacb3125c13a904a6c34d1c457a7e40e681526beaf046e03964e0236096"
Jul 17 21:55:56.492299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:56.492281    2563 scope.go:117] "RemoveContainer" containerID="ee90260e9de8159e710eb2679205c72790b2098d64fe5cba9e30a801137389e4"
Jul 17 21:55:56.492510 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:55:56.492489    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx_odh-ai-gateway-infra(3b6a3bf8-c928-4bc0-93f2-201e302605fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd"
Jul 17 21:55:57.518617 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:57.518566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"]
Jul 17 21:55:57.658304 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:57.658277    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:57.781362 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:57.781274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6a3bf8-c928-4bc0-93f2-201e302605fd-maas-api-service-ca\") pod \"3b6a3bf8-c928-4bc0-93f2-201e302605fd\" (UID: \"3b6a3bf8-c928-4bc0-93f2-201e302605fd\") "
Jul 17 21:55:57.781719 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:57.781690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6a3bf8-c928-4bc0-93f2-201e302605fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b6a3bf8-c928-4bc0-93f2-201e302605fd" (UID: "3b6a3bf8-c928-4bc0-93f2-201e302605fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:57.882266 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:57.882235    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6a3bf8-c928-4bc0-93f2-201e302605fd-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:55:58.504653 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:58.504592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx" event={"ID":"3b6a3bf8-c928-4bc0-93f2-201e302605fd","Type":"ContainerDied","Data":"a1be577e4fc92ac852302d52ddd27794e7b68ddec44408d9c95ce3b669f575de"}
Jul 17 21:55:58.504841 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:58.504675    2563 scope.go:117] "RemoveContainer" containerID="ee90260e9de8159e710eb2679205c72790b2098d64fe5cba9e30a801137389e4"
Jul 17 21:55:58.504841 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:58.504621    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"
Jul 17 21:55:58.525435 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:58.525406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"]
Jul 17 21:55:58.533148 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:58.533122    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-bclpx"]
Jul 17 21:55:58.602484 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:55:58.602450    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" path="/var/lib/kubelet/pods/3b6a3bf8-c928-4bc0-93f2-201e302605fd/volumes"
Jul 17 21:56:29.253558 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253525    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"]
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253905    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253916    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253930    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253935    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d6529a-7ea2-4d97-86d9-9680ec763f6c" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253944    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.253950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.254007    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.254016    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.254029 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.254022    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:29.256958 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.256941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:29.259379 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.259362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:56:29.265614 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.265575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"]
Jul 17 21:56:29.363455 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.363426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2f7898-da91-4beb-b2d9-a73cc84be051-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl\" (UID: \"ed2f7898-da91-4beb-b2d9-a73cc84be051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:29.464930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.464891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2f7898-da91-4beb-b2d9-a73cc84be051-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl\" (UID: \"ed2f7898-da91-4beb-b2d9-a73cc84be051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:29.465624 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.465575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2f7898-da91-4beb-b2d9-a73cc84be051-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl\" (UID: \"ed2f7898-da91-4beb-b2d9-a73cc84be051\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:29.568734 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.568645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:29.707125 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:29.707046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"]
Jul 17 21:56:29.707629 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:56:29.707576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2f7898_da91_4beb_b2d9_a73cc84be051.slice/crio-875799fff89435630c68f2c3f66cb6458c9032de51211d7babad948f2d7c5e13 WatchSource:0}: Error finding container 875799fff89435630c68f2c3f66cb6458c9032de51211d7babad948f2d7c5e13: Status 404 returned error can't find the container with id 875799fff89435630c68f2c3f66cb6458c9032de51211d7babad948f2d7c5e13
Jul 17 21:56:30.655300 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:30.655219    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerID="8fe4e00e1237e7d304956f9b4d246e0aaca7754896b7725ffdc6d181659907b0" exitCode=7
Jul 17 21:56:30.655711 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:30.655308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" event={"ID":"ed2f7898-da91-4beb-b2d9-a73cc84be051","Type":"ContainerDied","Data":"8fe4e00e1237e7d304956f9b4d246e0aaca7754896b7725ffdc6d181659907b0"}
Jul 17 21:56:30.655711 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:30.655362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" event={"ID":"ed2f7898-da91-4beb-b2d9-a73cc84be051","Type":"ContainerStarted","Data":"875799fff89435630c68f2c3f66cb6458c9032de51211d7babad948f2d7c5e13"}
Jul 17 21:56:30.655711 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:30.655484    2563 scope.go:117] "RemoveContainer" containerID="8fe4e00e1237e7d304956f9b4d246e0aaca7754896b7725ffdc6d181659907b0"
Jul 17 21:56:31.662197 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:31.662165    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerID="6fe8cf03625cccea2eecf832468efa77d942a19db738bfb0b333ee3475a72542" exitCode=7
Jul 17 21:56:31.662615 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:31.662221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" event={"ID":"ed2f7898-da91-4beb-b2d9-a73cc84be051","Type":"ContainerDied","Data":"6fe8cf03625cccea2eecf832468efa77d942a19db738bfb0b333ee3475a72542"}
Jul 17 21:56:31.662615 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:31.662280    2563 scope.go:117] "RemoveContainer" containerID="8fe4e00e1237e7d304956f9b4d246e0aaca7754896b7725ffdc6d181659907b0"
Jul 17 21:56:31.662615 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:31.662491    2563 scope.go:117] "RemoveContainer" containerID="6fe8cf03625cccea2eecf832468efa77d942a19db738bfb0b333ee3475a72542"
Jul 17 21:56:31.662743 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:56:31.662724    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl_odh-ai-gateway-infra(ed2f7898-da91-4beb-b2d9-a73cc84be051)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051"
Jul 17 21:56:32.668874 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:32.668848    2563 scope.go:117] "RemoveContainer" containerID="6fe8cf03625cccea2eecf832468efa77d942a19db738bfb0b333ee3475a72542"
Jul 17 21:56:32.669230 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:56:32.669028    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl_odh-ai-gateway-infra(ed2f7898-da91-4beb-b2d9-a73cc84be051)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051"
Jul 17 21:56:44.596649 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:44.596614    2563 scope.go:117] "RemoveContainer" containerID="6fe8cf03625cccea2eecf832468efa77d942a19db738bfb0b333ee3475a72542"
Jul 17 21:56:45.729457 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:45.729420    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerID="d100721bd307bee806d94d9fbeca60e51c59bc0b4dbfd517de63a9a73063e636" exitCode=6
Jul 17 21:56:45.729956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:45.729488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" event={"ID":"ed2f7898-da91-4beb-b2d9-a73cc84be051","Type":"ContainerDied","Data":"d100721bd307bee806d94d9fbeca60e51c59bc0b4dbfd517de63a9a73063e636"}
Jul 17 21:56:45.729956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:45.729567    2563 scope.go:117] "RemoveContainer" containerID="6fe8cf03625cccea2eecf832468efa77d942a19db738bfb0b333ee3475a72542"
Jul 17 21:56:45.729956 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:45.729922    2563 scope.go:117] "RemoveContainer" containerID="d100721bd307bee806d94d9fbeca60e51c59bc0b4dbfd517de63a9a73063e636"
Jul 17 21:56:45.730202 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:56:45.730172    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl_odh-ai-gateway-infra(ed2f7898-da91-4beb-b2d9-a73cc84be051)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051"
Jul 17 21:56:46.754509 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:46.754477    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"]
Jul 17 21:56:46.896256 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:46.896230    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:47.003361 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"]
Jul 17 21:56:47.003701 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003689    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:56:47.003755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:56:47.003755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:47.003755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6a3bf8-c928-4bc0-93f2-201e302605fd" containerName="revoke-keys"
Jul 17 21:56:47.003755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:56:47.003755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:56:47.003902 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:56:47.003902 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.003807    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:56:47.006679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.006622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:56:47.015291 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.015267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"]
Jul 17 21:56:47.016920 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.016901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2f7898-da91-4beb-b2d9-a73cc84be051-maas-api-service-ca\") pod \"ed2f7898-da91-4beb-b2d9-a73cc84be051\" (UID: \"ed2f7898-da91-4beb-b2d9-a73cc84be051\") "
Jul 17 21:56:47.017279 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.017251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2f7898-da91-4beb-b2d9-a73cc84be051-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed2f7898-da91-4beb-b2d9-a73cc84be051" (UID: "ed2f7898-da91-4beb-b2d9-a73cc84be051"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:47.118680 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.118644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f4af68-8460-4c0d-8786-633df7953528-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2\" (UID: \"14f4af68-8460-4c0d-8786-633df7953528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:56:47.118863 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.118716    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2f7898-da91-4beb-b2d9-a73cc84be051-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:56:47.219296 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.219260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f4af68-8460-4c0d-8786-633df7953528-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2\" (UID: \"14f4af68-8460-4c0d-8786-633df7953528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:56:47.219905 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.219886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f4af68-8460-4c0d-8786-633df7953528-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2\" (UID: \"14f4af68-8460-4c0d-8786-633df7953528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:56:47.319223 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.319144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:56:47.452961 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.452936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"]
Jul 17 21:56:47.455289 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:56:47.455255    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f4af68_8460_4c0d_8786_633df7953528.slice/crio-52aaf202bcd6c3786c8ea7ca918d84e290ed839fc8e1584f0ac041e5a0ec353e WatchSource:0}: Error finding container 52aaf202bcd6c3786c8ea7ca918d84e290ed839fc8e1584f0ac041e5a0ec353e: Status 404 returned error can't find the container with id 52aaf202bcd6c3786c8ea7ca918d84e290ed839fc8e1584f0ac041e5a0ec353e
Jul 17 21:56:47.743591 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.743559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" event={"ID":"14f4af68-8460-4c0d-8786-633df7953528","Type":"ContainerStarted","Data":"87c7f161c12a7a0ca39ca4838b6b5a12c2527b7c1de2d3b0a4365f115982a021"}
Jul 17 21:56:47.743766 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.743649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" event={"ID":"14f4af68-8460-4c0d-8786-633df7953528","Type":"ContainerStarted","Data":"52aaf202bcd6c3786c8ea7ca918d84e290ed839fc8e1584f0ac041e5a0ec353e"}
Jul 17 21:56:47.744732 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.744707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl" event={"ID":"ed2f7898-da91-4beb-b2d9-a73cc84be051","Type":"ContainerDied","Data":"875799fff89435630c68f2c3f66cb6458c9032de51211d7babad948f2d7c5e13"}
Jul 17 21:56:47.744833 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.744738    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"
Jul 17 21:56:47.744833 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.744764    2563 scope.go:117] "RemoveContainer" containerID="d100721bd307bee806d94d9fbeca60e51c59bc0b4dbfd517de63a9a73063e636"
Jul 17 21:56:47.758545 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.758490    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" podStartSLOduration=1.758470609 podStartE2EDuration="1.758470609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:47.758371738 +0000 UTC m=+1771.670612020" watchObservedRunningTime="2026-07-17 21:56:47.758470609 +0000 UTC m=+1771.670710898"
Jul 17 21:56:47.772420 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.772395    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"]
Jul 17 21:56:47.776131 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:47.776110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fgpnl"]
Jul 17 21:56:48.602132 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:48.602099    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" path="/var/lib/kubelet/pods/ed2f7898-da91-4beb-b2d9-a73cc84be051/volumes"
Jul 17 21:56:48.751551 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:48.751520    2563 generic.go:356] "Generic (PLEG): container finished" podID="14f4af68-8460-4c0d-8786-633df7953528" containerID="87c7f161c12a7a0ca39ca4838b6b5a12c2527b7c1de2d3b0a4365f115982a021" exitCode=7
Jul 17 21:56:48.751551 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:48.751556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" event={"ID":"14f4af68-8460-4c0d-8786-633df7953528","Type":"ContainerDied","Data":"87c7f161c12a7a0ca39ca4838b6b5a12c2527b7c1de2d3b0a4365f115982a021"}
Jul 17 21:56:48.751778 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:48.751764    2563 scope.go:117] "RemoveContainer" containerID="87c7f161c12a7a0ca39ca4838b6b5a12c2527b7c1de2d3b0a4365f115982a021"
Jul 17 21:56:49.757895 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:49.757856    2563 generic.go:356] "Generic (PLEG): container finished" podID="14f4af68-8460-4c0d-8786-633df7953528" containerID="fa84ca0bafc5131ba610da7983f89deb96a6e9de380c393b7fd24d1334be6bf4" exitCode=7
Jul 17 21:56:49.758280 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:49.757947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" event={"ID":"14f4af68-8460-4c0d-8786-633df7953528","Type":"ContainerDied","Data":"fa84ca0bafc5131ba610da7983f89deb96a6e9de380c393b7fd24d1334be6bf4"}
Jul 17 21:56:49.758280 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:49.758011    2563 scope.go:117] "RemoveContainer" containerID="87c7f161c12a7a0ca39ca4838b6b5a12c2527b7c1de2d3b0a4365f115982a021"
Jul 17 21:56:49.758280 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:49.758255    2563 scope.go:117] "RemoveContainer" containerID="fa84ca0bafc5131ba610da7983f89deb96a6e9de380c393b7fd24d1334be6bf4"
Jul 17 21:56:49.758501 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:56:49.758477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2_odh-ai-gateway-infra(14f4af68-8460-4c0d-8786-633df7953528)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" podUID="14f4af68-8460-4c0d-8786-633df7953528"
Jul 17 21:56:50.764561 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:56:50.764535    2563 scope.go:117] "RemoveContainer" containerID="fa84ca0bafc5131ba610da7983f89deb96a6e9de380c393b7fd24d1334be6bf4"
Jul 17 21:56:50.764963 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:56:50.764762    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2_odh-ai-gateway-infra(14f4af68-8460-4c0d-8786-633df7953528)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" podUID="14f4af68-8460-4c0d-8786-633df7953528"
Jul 17 21:57:01.596731 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:01.596699    2563 scope.go:117] "RemoveContainer" containerID="fa84ca0bafc5131ba610da7983f89deb96a6e9de380c393b7fd24d1334be6bf4"
Jul 17 21:57:02.824890 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:02.824856    2563 generic.go:356] "Generic (PLEG): container finished" podID="14f4af68-8460-4c0d-8786-633df7953528" containerID="9f1fa829ce6694891df2e73152f0ae35d4ea1e51157ac14ddaa762b5bc24e1d4" exitCode=7
Jul 17 21:57:02.825287 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:02.824925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" event={"ID":"14f4af68-8460-4c0d-8786-633df7953528","Type":"ContainerDied","Data":"9f1fa829ce6694891df2e73152f0ae35d4ea1e51157ac14ddaa762b5bc24e1d4"}
Jul 17 21:57:02.825287 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:02.824985    2563 scope.go:117] "RemoveContainer" containerID="fa84ca0bafc5131ba610da7983f89deb96a6e9de380c393b7fd24d1334be6bf4"
Jul 17 21:57:02.825287 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:02.825265    2563 scope.go:117] "RemoveContainer" containerID="9f1fa829ce6694891df2e73152f0ae35d4ea1e51157ac14ddaa762b5bc24e1d4"
Jul 17 21:57:02.825477 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:02.825455    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2_odh-ai-gateway-infra(14f4af68-8460-4c0d-8786-633df7953528)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" podUID="14f4af68-8460-4c0d-8786-633df7953528"
Jul 17 21:57:03.849440 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:03.849404    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"]
Jul 17 21:57:03.988028 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:03.988001    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:57:04.068189 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.068158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f4af68-8460-4c0d-8786-633df7953528-maas-api-service-ca\") pod \"14f4af68-8460-4c0d-8786-633df7953528\" (UID: \"14f4af68-8460-4c0d-8786-633df7953528\") "
Jul 17 21:57:04.068569 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.068546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f4af68-8460-4c0d-8786-633df7953528-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14f4af68-8460-4c0d-8786-633df7953528" (UID: "14f4af68-8460-4c0d-8786-633df7953528"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:04.169889 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.169854    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f4af68-8460-4c0d-8786-633df7953528-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:57:04.837708 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.837671    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"
Jul 17 21:57:04.837708 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.837687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2" event={"ID":"14f4af68-8460-4c0d-8786-633df7953528","Type":"ContainerDied","Data":"52aaf202bcd6c3786c8ea7ca918d84e290ed839fc8e1584f0ac041e5a0ec353e"}
Jul 17 21:57:04.837917 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.837751    2563 scope.go:117] "RemoveContainer" containerID="9f1fa829ce6694891df2e73152f0ae35d4ea1e51157ac14ddaa762b5bc24e1d4"
Jul 17 21:57:04.852750 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.852719    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"]
Jul 17 21:57:04.859211 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:04.859188    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-v5hx2"]
Jul 17 21:57:06.604211 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:06.604171    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f4af68-8460-4c0d-8786-633df7953528" path="/var/lib/kubelet/pods/14f4af68-8460-4c0d-8786-633df7953528/volumes"
Jul 17 21:57:16.666385 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:16.666357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:57:16.668554 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:16.668526    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 21:57:19.293873 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.293836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"]
Jul 17 21:57:19.294239 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294207    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294239 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294218    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294239 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294229    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294239 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294235    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294255    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294269    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294275    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294333    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294342    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294347    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2f7898-da91-4beb-b2d9-a73cc84be051" containerName="revoke-keys"
Jul 17 21:57:19.294381 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.294355    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f4af68-8460-4c0d-8786-633df7953528" containerName="revoke-keys"
Jul 17 21:57:19.297337 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.297319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:19.299691 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.299666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:57:19.303917 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.303886    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"]
Jul 17 21:57:19.409454 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.409415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05211ce9-177e-41d3-9920-3c62e5b6c941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq\" (UID: \"05211ce9-177e-41d3-9920-3c62e5b6c941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:19.509851 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.509819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05211ce9-177e-41d3-9920-3c62e5b6c941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq\" (UID: \"05211ce9-177e-41d3-9920-3c62e5b6c941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:19.510427 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.510406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05211ce9-177e-41d3-9920-3c62e5b6c941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq\" (UID: \"05211ce9-177e-41d3-9920-3c62e5b6c941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:19.609124 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.609047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:19.739213 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.739182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"]
Jul 17 21:57:19.741088 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:57:19.741051    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05211ce9_177e_41d3_9920_3c62e5b6c941.slice/crio-f01a664d568c60562f1e2cb3adbec1d13289c3741fd93eb8ca3a2a081e95c06b WatchSource:0}: Error finding container f01a664d568c60562f1e2cb3adbec1d13289c3741fd93eb8ca3a2a081e95c06b: Status 404 returned error can't find the container with id f01a664d568c60562f1e2cb3adbec1d13289c3741fd93eb8ca3a2a081e95c06b
Jul 17 21:57:19.910127 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:19.910041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" event={"ID":"05211ce9-177e-41d3-9920-3c62e5b6c941","Type":"ContainerStarted","Data":"f01a664d568c60562f1e2cb3adbec1d13289c3741fd93eb8ca3a2a081e95c06b"}
Jul 17 21:57:20.917303 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:20.917262    2563 generic.go:356] "Generic (PLEG): container finished" podID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerID="c82fe531f4a875998fa0e25ae836c44e7380675d218db139eb8441307c4adaff" exitCode=6
Jul 17 21:57:20.917788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:20.917340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" event={"ID":"05211ce9-177e-41d3-9920-3c62e5b6c941","Type":"ContainerDied","Data":"c82fe531f4a875998fa0e25ae836c44e7380675d218db139eb8441307c4adaff"}
Jul 17 21:57:20.917788 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:20.917557    2563 scope.go:117] "RemoveContainer" containerID="c82fe531f4a875998fa0e25ae836c44e7380675d218db139eb8441307c4adaff"
Jul 17 21:57:21.924052 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:21.924017    2563 generic.go:356] "Generic (PLEG): container finished" podID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerID="a94b0c02670d322147fbc4c2276da45ed52f4971395fa8dd7916782a4d334229" exitCode=6
Jul 17 21:57:21.924449 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:21.924108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" event={"ID":"05211ce9-177e-41d3-9920-3c62e5b6c941","Type":"ContainerDied","Data":"a94b0c02670d322147fbc4c2276da45ed52f4971395fa8dd7916782a4d334229"}
Jul 17 21:57:21.924449 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:21.924216    2563 scope.go:117] "RemoveContainer" containerID="c82fe531f4a875998fa0e25ae836c44e7380675d218db139eb8441307c4adaff"
Jul 17 21:57:21.924449 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:21.924281    2563 scope.go:117] "RemoveContainer" containerID="a94b0c02670d322147fbc4c2276da45ed52f4971395fa8dd7916782a4d334229"
Jul 17 21:57:21.924639 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:21.924490    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq_odh-ai-gateway-infra(05211ce9-177e-41d3-9920-3c62e5b6c941)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941"
Jul 17 21:57:22.931036 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:22.931013    2563 scope.go:117] "RemoveContainer" containerID="a94b0c02670d322147fbc4c2276da45ed52f4971395fa8dd7916782a4d334229"
Jul 17 21:57:22.931443 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:22.931197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq_odh-ai-gateway-infra(05211ce9-177e-41d3-9920-3c62e5b6c941)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941"
Jul 17 21:57:36.598906 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:36.598866    2563 scope.go:117] "RemoveContainer" containerID="a94b0c02670d322147fbc4c2276da45ed52f4971395fa8dd7916782a4d334229"
Jul 17 21:57:36.996249 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:36.996212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" event={"ID":"05211ce9-177e-41d3-9920-3c62e5b6c941","Type":"ContainerStarted","Data":"e1152d7d2722b2b6735de8cfcc5ede58968150e2944c710697a13bff98e687be"}
Jul 17 21:57:37.012151 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.012105    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" podStartSLOduration=18.012092309 podStartE2EDuration="18.012092309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:37.011045379 +0000 UTC m=+1820.923285668" watchObservedRunningTime="2026-07-17 21:57:37.012092309 +0000 UTC m=+1820.924332598"
Jul 17 21:57:37.044150 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.044120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"]
Jul 17 21:57:37.047591 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.047575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:37.055415 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.055385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"]
Jul 17 21:57:37.163893 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.163862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cdf4359-d8e1-4862-bd4c-211b37c6ced8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9\" (UID: \"1cdf4359-d8e1-4862-bd4c-211b37c6ced8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:37.264359 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.264330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cdf4359-d8e1-4862-bd4c-211b37c6ced8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9\" (UID: \"1cdf4359-d8e1-4862-bd4c-211b37c6ced8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:37.264963 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.264940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cdf4359-d8e1-4862-bd4c-211b37c6ced8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9\" (UID: \"1cdf4359-d8e1-4862-bd4c-211b37c6ced8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:37.358883 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.358852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:37.495976 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:37.495947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"]
Jul 17 21:57:37.498412 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:57:37.498385    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdf4359_d8e1_4862_bd4c_211b37c6ced8.slice/crio-c8eabed8b3a8ad13594a2e49fbdecb4c3de2041b340adfe6b753da5b50ceb57d WatchSource:0}: Error finding container c8eabed8b3a8ad13594a2e49fbdecb4c3de2041b340adfe6b753da5b50ceb57d: Status 404 returned error can't find the container with id c8eabed8b3a8ad13594a2e49fbdecb4c3de2041b340adfe6b753da5b50ceb57d
Jul 17 21:57:38.003184 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.003151    2563 generic.go:356] "Generic (PLEG): container finished" podID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerID="e1152d7d2722b2b6735de8cfcc5ede58968150e2944c710697a13bff98e687be" exitCode=6
Jul 17 21:57:38.003628 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.003227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" event={"ID":"05211ce9-177e-41d3-9920-3c62e5b6c941","Type":"ContainerDied","Data":"e1152d7d2722b2b6735de8cfcc5ede58968150e2944c710697a13bff98e687be"}
Jul 17 21:57:38.003628 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.003296    2563 scope.go:117] "RemoveContainer" containerID="a94b0c02670d322147fbc4c2276da45ed52f4971395fa8dd7916782a4d334229"
Jul 17 21:57:38.003628 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.003591    2563 scope.go:117] "RemoveContainer" containerID="e1152d7d2722b2b6735de8cfcc5ede58968150e2944c710697a13bff98e687be"
Jul 17 21:57:38.003860 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:38.003832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq_odh-ai-gateway-infra(05211ce9-177e-41d3-9920-3c62e5b6c941)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941"
Jul 17 21:57:38.004935 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.004877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerStarted","Data":"65b3d47032df815fa684132cfbad59c199fe992f0a6b7af10546f8c654c5f0fa"}
Jul 17 21:57:38.005032 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.004961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerStarted","Data":"c8eabed8b3a8ad13594a2e49fbdecb4c3de2041b340adfe6b753da5b50ceb57d"}
Jul 17 21:57:38.031384 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.031358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"]
Jul 17 21:57:38.036151 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:38.036109    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" podStartSLOduration=1.036096122 podStartE2EDuration="1.036096122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:38.034855871 +0000 UTC m=+1821.947096180" watchObservedRunningTime="2026-07-17 21:57:38.036096122 +0000 UTC m=+1821.948336411"
Jul 17 21:57:39.012664 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.012627    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerID="65b3d47032df815fa684132cfbad59c199fe992f0a6b7af10546f8c654c5f0fa" exitCode=7
Jul 17 21:57:39.012986 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.012685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerDied","Data":"65b3d47032df815fa684132cfbad59c199fe992f0a6b7af10546f8c654c5f0fa"}
Jul 17 21:57:39.012986 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.012915    2563 scope.go:117] "RemoveContainer" containerID="65b3d47032df815fa684132cfbad59c199fe992f0a6b7af10546f8c654c5f0fa"
Jul 17 21:57:39.157717 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.157697    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:39.283478 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.283396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05211ce9-177e-41d3-9920-3c62e5b6c941-maas-api-service-ca\") pod \"05211ce9-177e-41d3-9920-3c62e5b6c941\" (UID: \"05211ce9-177e-41d3-9920-3c62e5b6c941\") "
Jul 17 21:57:39.283789 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.283766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05211ce9-177e-41d3-9920-3c62e5b6c941-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05211ce9-177e-41d3-9920-3c62e5b6c941" (UID: "05211ce9-177e-41d3-9920-3c62e5b6c941"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:39.385076 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:39.385041    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05211ce9-177e-41d3-9920-3c62e5b6c941-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.019035 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.018996    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"
Jul 17 21:57:40.019502 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.019001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq" event={"ID":"05211ce9-177e-41d3-9920-3c62e5b6c941","Type":"ContainerDied","Data":"f01a664d568c60562f1e2cb3adbec1d13289c3741fd93eb8ca3a2a081e95c06b"}
Jul 17 21:57:40.019502 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.019150    2563 scope.go:117] "RemoveContainer" containerID="e1152d7d2722b2b6735de8cfcc5ede58968150e2944c710697a13bff98e687be"
Jul 17 21:57:40.020999 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.020969    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerID="a4264793b47f95d67eb31a5932ae0f4e0f5d655cce2ef1fad50faf8fabdf89d3" exitCode=7
Jul 17 21:57:40.021114 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.021040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerDied","Data":"a4264793b47f95d67eb31a5932ae0f4e0f5d655cce2ef1fad50faf8fabdf89d3"}
Jul 17 21:57:40.021264 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.021249    2563 scope.go:117] "RemoveContainer" containerID="a4264793b47f95d67eb31a5932ae0f4e0f5d655cce2ef1fad50faf8fabdf89d3"
Jul 17 21:57:40.021474 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:40.021458    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9_odh-ai-gateway-infra(1cdf4359-d8e1-4862-bd4c-211b37c6ced8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8"
Jul 17 21:57:40.028409 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.028390    2563 scope.go:117] "RemoveContainer" containerID="65b3d47032df815fa684132cfbad59c199fe992f0a6b7af10546f8c654c5f0fa"
Jul 17 21:57:40.053219 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.053191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"]
Jul 17 21:57:40.057172 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.057149    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4n6gq"]
Jul 17 21:57:40.605445 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:40.605412    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" path="/var/lib/kubelet/pods/05211ce9-177e-41d3-9920-3c62e5b6c941/volumes"
Jul 17 21:57:41.027877 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:41.027847    2563 scope.go:117] "RemoveContainer" containerID="a4264793b47f95d67eb31a5932ae0f4e0f5d655cce2ef1fad50faf8fabdf89d3"
Jul 17 21:57:41.028270 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:41.028056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9_odh-ai-gateway-infra(1cdf4359-d8e1-4862-bd4c-211b37c6ced8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8"
Jul 17 21:57:51.597333 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:51.597304    2563 scope.go:117] "RemoveContainer" containerID="a4264793b47f95d67eb31a5932ae0f4e0f5d655cce2ef1fad50faf8fabdf89d3"
Jul 17 21:57:52.079662 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:52.079629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerStarted","Data":"28dc8ff6c968c15116aabc3388ba451a94d40c420d7d717637ea659a8e3c4ca2"}
Jul 17 21:57:53.085620 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:53.085567    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerID="28dc8ff6c968c15116aabc3388ba451a94d40c420d7d717637ea659a8e3c4ca2" exitCode=7
Jul 17 21:57:53.086068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:53.085644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerDied","Data":"28dc8ff6c968c15116aabc3388ba451a94d40c420d7d717637ea659a8e3c4ca2"}
Jul 17 21:57:53.086068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:53.085720    2563 scope.go:117] "RemoveContainer" containerID="a4264793b47f95d67eb31a5932ae0f4e0f5d655cce2ef1fad50faf8fabdf89d3"
Jul 17 21:57:53.086068 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:53.086026    2563 scope.go:117] "RemoveContainer" containerID="28dc8ff6c968c15116aabc3388ba451a94d40c420d7d717637ea659a8e3c4ca2"
Jul 17 21:57:53.086261 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:57:53.086243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9_odh-ai-gateway-infra(1cdf4359-d8e1-4862-bd4c-211b37c6ced8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8"
Jul 17 21:57:53.109382 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:53.109354    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"]
Jul 17 21:57:54.234134 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:54.234110    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:54.315699 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:54.315663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cdf4359-d8e1-4862-bd4c-211b37c6ced8-maas-api-service-ca\") pod \"1cdf4359-d8e1-4862-bd4c-211b37c6ced8\" (UID: \"1cdf4359-d8e1-4862-bd4c-211b37c6ced8\") "
Jul 17 21:57:54.316017 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:54.315991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdf4359-d8e1-4862-bd4c-211b37c6ced8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cdf4359-d8e1-4862-bd4c-211b37c6ced8" (UID: "1cdf4359-d8e1-4862-bd4c-211b37c6ced8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:54.417173 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:54.417145    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cdf4359-d8e1-4862-bd4c-211b37c6ced8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:57:55.097662 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:55.097637    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"
Jul 17 21:57:55.097837 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:55.097636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9" event={"ID":"1cdf4359-d8e1-4862-bd4c-211b37c6ced8","Type":"ContainerDied","Data":"c8eabed8b3a8ad13594a2e49fbdecb4c3de2041b340adfe6b753da5b50ceb57d"}
Jul 17 21:57:55.097837 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:55.097770    2563 scope.go:117] "RemoveContainer" containerID="28dc8ff6c968c15116aabc3388ba451a94d40c420d7d717637ea659a8e3c4ca2"
Jul 17 21:57:55.116222 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:55.116189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"]
Jul 17 21:57:55.123789 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:55.123764    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zqrv9"]
Jul 17 21:57:56.604265 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:57:56.604236    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" path="/var/lib/kubelet/pods/1cdf4359-d8e1-4862-bd4c-211b37c6ced8/volumes"
Jul 17 21:58:04.992544 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.992500    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"]
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993057    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993073    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993095    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993103    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993122    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993144    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993152    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993267    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993276    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993286    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:04.993259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:04.993298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:05.009725 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.009690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"]
Jul 17 21:58:05.009907 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.009785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:05.012480 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.012456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:58:05.111369 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.111335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7922414b-5fa7-4aa8-b002-a945a3c85c36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv\" (UID: \"7922414b-5fa7-4aa8-b002-a945a3c85c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:05.211888 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.211855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7922414b-5fa7-4aa8-b002-a945a3c85c36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv\" (UID: \"7922414b-5fa7-4aa8-b002-a945a3c85c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:05.212465 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.212446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7922414b-5fa7-4aa8-b002-a945a3c85c36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv\" (UID: \"7922414b-5fa7-4aa8-b002-a945a3c85c36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:05.324896 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.324803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:05.461675 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:05.461521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"]
Jul 17 21:58:05.464864 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:58:05.464832    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7922414b_5fa7_4aa8_b002_a945a3c85c36.slice/crio-d663cc530dbeb2edb11d779e9e61d11d1466bf8777aa662399e3c51f95698b7b WatchSource:0}: Error finding container d663cc530dbeb2edb11d779e9e61d11d1466bf8777aa662399e3c51f95698b7b: Status 404 returned error can't find the container with id d663cc530dbeb2edb11d779e9e61d11d1466bf8777aa662399e3c51f95698b7b
Jul 17 21:58:06.152571 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:06.152481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerStarted","Data":"8669e7e66f55d06c4f07825ebe2fe1b736bc402ed4f0f74a0ff41f1cbef2e85c"}
Jul 17 21:58:06.152571 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:06.152533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerStarted","Data":"d663cc530dbeb2edb11d779e9e61d11d1466bf8777aa662399e3c51f95698b7b"}
Jul 17 21:58:06.168568 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:06.168462    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" podStartSLOduration=2.168446563 podStartE2EDuration="2.168446563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:06.167320468 +0000 UTC m=+1850.079560757" watchObservedRunningTime="2026-07-17 21:58:06.168446563 +0000 UTC m=+1850.080686851"
Jul 17 21:58:07.158404 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:07.158369    2563 generic.go:356] "Generic (PLEG): container finished" podID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerID="8669e7e66f55d06c4f07825ebe2fe1b736bc402ed4f0f74a0ff41f1cbef2e85c" exitCode=7
Jul 17 21:58:07.158857 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:07.158436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerDied","Data":"8669e7e66f55d06c4f07825ebe2fe1b736bc402ed4f0f74a0ff41f1cbef2e85c"}
Jul 17 21:58:07.158857 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:07.158648    2563 scope.go:117] "RemoveContainer" containerID="8669e7e66f55d06c4f07825ebe2fe1b736bc402ed4f0f74a0ff41f1cbef2e85c"
Jul 17 21:58:08.164607 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:08.164561    2563 generic.go:356] "Generic (PLEG): container finished" podID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerID="60ac0b9bb1af4f1c093fea697629746e3eab3c5e71e0db1f92ddd75d1fed98f5" exitCode=7
Jul 17 21:58:08.165078 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:08.164647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerDied","Data":"60ac0b9bb1af4f1c093fea697629746e3eab3c5e71e0db1f92ddd75d1fed98f5"}
Jul 17 21:58:08.165078 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:08.164705    2563 scope.go:117] "RemoveContainer" containerID="8669e7e66f55d06c4f07825ebe2fe1b736bc402ed4f0f74a0ff41f1cbef2e85c"
Jul 17 21:58:08.165078 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:08.164907    2563 scope.go:117] "RemoveContainer" containerID="60ac0b9bb1af4f1c093fea697629746e3eab3c5e71e0db1f92ddd75d1fed98f5"
Jul 17 21:58:08.165234 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:08.165173    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv_odh-ai-gateway-infra(7922414b-5fa7-4aa8-b002-a945a3c85c36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36"
Jul 17 21:58:09.171676 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:09.171645    2563 scope.go:117] "RemoveContainer" containerID="60ac0b9bb1af4f1c093fea697629746e3eab3c5e71e0db1f92ddd75d1fed98f5"
Jul 17 21:58:09.172097 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:09.171833    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv_odh-ai-gateway-infra(7922414b-5fa7-4aa8-b002-a945a3c85c36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36"
Jul 17 21:58:19.338981 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.338947    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"]
Jul 17 21:58:19.339369 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.339299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:19.339369 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.339310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:19.339369 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.339328    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:19.339369 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.339333    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:19.339503 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.339387    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05211ce9-177e-41d3-9920-3c62e5b6c941" containerName="revoke-keys"
Jul 17 21:58:19.339503 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.339394    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdf4359-d8e1-4862-bd4c-211b37c6ced8" containerName="revoke-keys"
Jul 17 21:58:19.342278 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.342261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:19.348972 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.348943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"]
Jul 17 21:58:19.433703 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.433670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e2db6d-0c32-40bd-84b7-e78fb6833306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-467f4\" (UID: \"66e2db6d-0c32-40bd-84b7-e78fb6833306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:19.535125 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.535086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e2db6d-0c32-40bd-84b7-e78fb6833306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-467f4\" (UID: \"66e2db6d-0c32-40bd-84b7-e78fb6833306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:19.535767 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.535749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e2db6d-0c32-40bd-84b7-e78fb6833306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-467f4\" (UID: \"66e2db6d-0c32-40bd-84b7-e78fb6833306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:19.653827 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.653746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:19.790618 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:19.790572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"]
Jul 17 21:58:19.792525 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:58:19.792487    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e2db6d_0c32_40bd_84b7_e78fb6833306.slice/crio-5baf66222febc7aad8ddd0eed31adcc96733378d3929d0148c189053efceded4 WatchSource:0}: Error finding container 5baf66222febc7aad8ddd0eed31adcc96733378d3929d0148c189053efceded4: Status 404 returned error can't find the container with id 5baf66222febc7aad8ddd0eed31adcc96733378d3929d0148c189053efceded4
Jul 17 21:58:20.225340 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:20.225255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" event={"ID":"66e2db6d-0c32-40bd-84b7-e78fb6833306","Type":"ContainerStarted","Data":"a85db38c610d9a63f473900758492ad39d383d00469f64c7ad1bcee2eef9d7bf"}
Jul 17 21:58:20.225340 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:20.225313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" event={"ID":"66e2db6d-0c32-40bd-84b7-e78fb6833306","Type":"ContainerStarted","Data":"5baf66222febc7aad8ddd0eed31adcc96733378d3929d0148c189053efceded4"}
Jul 17 21:58:20.240249 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:20.240187    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" podStartSLOduration=1.240170706 podStartE2EDuration="1.240170706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:20.238197736 +0000 UTC m=+1864.150438025" watchObservedRunningTime="2026-07-17 21:58:20.240170706 +0000 UTC m=+1864.152410996"
Jul 17 21:58:21.231722 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:21.231682    2563 generic.go:356] "Generic (PLEG): container finished" podID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerID="a85db38c610d9a63f473900758492ad39d383d00469f64c7ad1bcee2eef9d7bf" exitCode=6
Jul 17 21:58:21.232190 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:21.231767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" event={"ID":"66e2db6d-0c32-40bd-84b7-e78fb6833306","Type":"ContainerDied","Data":"a85db38c610d9a63f473900758492ad39d383d00469f64c7ad1bcee2eef9d7bf"}
Jul 17 21:58:21.232190 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:21.231959    2563 scope.go:117] "RemoveContainer" containerID="a85db38c610d9a63f473900758492ad39d383d00469f64c7ad1bcee2eef9d7bf"
Jul 17 21:58:21.598033 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:21.597659    2563 scope.go:117] "RemoveContainer" containerID="60ac0b9bb1af4f1c093fea697629746e3eab3c5e71e0db1f92ddd75d1fed98f5"
Jul 17 21:58:22.238441 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:22.238405    2563 generic.go:356] "Generic (PLEG): container finished" podID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerID="58e76ac2cb60614ff52208db0a6cd3dc988e227a2e52263ac1060df245d2a04c" exitCode=6
Jul 17 21:58:22.238915 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:22.238458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" event={"ID":"66e2db6d-0c32-40bd-84b7-e78fb6833306","Type":"ContainerDied","Data":"58e76ac2cb60614ff52208db0a6cd3dc988e227a2e52263ac1060df245d2a04c"}
Jul 17 21:58:22.238915 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:22.238536    2563 scope.go:117] "RemoveContainer" containerID="a85db38c610d9a63f473900758492ad39d383d00469f64c7ad1bcee2eef9d7bf"
Jul 17 21:58:22.238915 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:22.238743    2563 scope.go:117] "RemoveContainer" containerID="58e76ac2cb60614ff52208db0a6cd3dc988e227a2e52263ac1060df245d2a04c"
Jul 17 21:58:22.239051 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:22.239012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-467f4_odh-ai-gateway-infra(66e2db6d-0c32-40bd-84b7-e78fb6833306)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306"
Jul 17 21:58:22.240387 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:22.240367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerStarted","Data":"45a7d7feab56df3a1a47126227b23389ef16a1497b9de1d03c6f40f082174568"}
Jul 17 21:58:23.248474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:23.248442    2563 generic.go:356] "Generic (PLEG): container finished" podID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerID="45a7d7feab56df3a1a47126227b23389ef16a1497b9de1d03c6f40f082174568" exitCode=7
Jul 17 21:58:23.248978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:23.248520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerDied","Data":"45a7d7feab56df3a1a47126227b23389ef16a1497b9de1d03c6f40f082174568"}
Jul 17 21:58:23.248978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:23.248589    2563 scope.go:117] "RemoveContainer" containerID="60ac0b9bb1af4f1c093fea697629746e3eab3c5e71e0db1f92ddd75d1fed98f5"
Jul 17 21:58:23.248978 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:23.248891    2563 scope.go:117] "RemoveContainer" containerID="45a7d7feab56df3a1a47126227b23389ef16a1497b9de1d03c6f40f082174568"
Jul 17 21:58:23.249168 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:23.249140    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv_odh-ai-gateway-infra(7922414b-5fa7-4aa8-b002-a945a3c85c36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36"
Jul 17 21:58:23.250430 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:23.250415    2563 scope.go:117] "RemoveContainer" containerID="58e76ac2cb60614ff52208db0a6cd3dc988e227a2e52263ac1060df245d2a04c"
Jul 17 21:58:23.250622 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:23.250583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-467f4_odh-ai-gateway-infra(66e2db6d-0c32-40bd-84b7-e78fb6833306)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306"
Jul 17 21:58:23.277786 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:23.277759    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"]
Jul 17 21:58:24.392259 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:24.392231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:24.477302 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:24.477260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7922414b-5fa7-4aa8-b002-a945a3c85c36-maas-api-service-ca\") pod \"7922414b-5fa7-4aa8-b002-a945a3c85c36\" (UID: \"7922414b-5fa7-4aa8-b002-a945a3c85c36\") "
Jul 17 21:58:24.477669 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:24.477645    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7922414b-5fa7-4aa8-b002-a945a3c85c36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7922414b-5fa7-4aa8-b002-a945a3c85c36" (UID: "7922414b-5fa7-4aa8-b002-a945a3c85c36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:24.578022 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:24.577926    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7922414b-5fa7-4aa8-b002-a945a3c85c36-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:58:25.261474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:25.261440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"
Jul 17 21:58:25.261474 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:25.261447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv" event={"ID":"7922414b-5fa7-4aa8-b002-a945a3c85c36","Type":"ContainerDied","Data":"d663cc530dbeb2edb11d779e9e61d11d1466bf8777aa662399e3c51f95698b7b"}
Jul 17 21:58:25.261757 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:25.261543    2563 scope.go:117] "RemoveContainer" containerID="45a7d7feab56df3a1a47126227b23389ef16a1497b9de1d03c6f40f082174568"
Jul 17 21:58:25.277345 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:25.277318    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"]
Jul 17 21:58:25.282543 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:25.282522    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-92vlv"]
Jul 17 21:58:26.603475 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:26.603439    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" path="/var/lib/kubelet/pods/7922414b-5fa7-4aa8-b002-a945a3c85c36/volumes"
Jul 17 21:58:27.095138 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095105    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"]
Jul 17 21:58:27.095484 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095534 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095486    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095534 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095497    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095534 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095505    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095534 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095511    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095534 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095516    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095710 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095581    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.095710 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.095587    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:58:27.099963 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.099946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:27.106122 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.106094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"]
Jul 17 21:58:27.200872 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.200833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6af6f12-065d-4cb4-b06e-a773eeb85438-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559\" (UID: \"c6af6f12-065d-4cb4-b06e-a773eeb85438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:27.301867 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.301833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6af6f12-065d-4cb4-b06e-a773eeb85438-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559\" (UID: \"c6af6f12-065d-4cb4-b06e-a773eeb85438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:27.302509 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.302491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6af6f12-065d-4cb4-b06e-a773eeb85438-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559\" (UID: \"c6af6f12-065d-4cb4-b06e-a773eeb85438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:27.412152 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.412125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:27.548806 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:27.548782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"]
Jul 17 21:58:27.550889 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:58:27.550860    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6af6f12_065d_4cb4_b06e_a773eeb85438.slice/crio-2577935fd060bc78f4e6956e2cccecdccdeaaaf593b063a6b8ba03c9a323ddc9 WatchSource:0}: Error finding container 2577935fd060bc78f4e6956e2cccecdccdeaaaf593b063a6b8ba03c9a323ddc9: Status 404 returned error can't find the container with id 2577935fd060bc78f4e6956e2cccecdccdeaaaf593b063a6b8ba03c9a323ddc9
Jul 17 21:58:28.280764 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:28.280737    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerID="443ddb6c9cec6f04a7f838ec9a7bfe51841bc263f4b29d841122e182b03cac4f" exitCode=6
Jul 17 21:58:28.281090 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:28.280782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" event={"ID":"c6af6f12-065d-4cb4-b06e-a773eeb85438","Type":"ContainerDied","Data":"443ddb6c9cec6f04a7f838ec9a7bfe51841bc263f4b29d841122e182b03cac4f"}
Jul 17 21:58:28.281090 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:28.280825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" event={"ID":"c6af6f12-065d-4cb4-b06e-a773eeb85438","Type":"ContainerStarted","Data":"2577935fd060bc78f4e6956e2cccecdccdeaaaf593b063a6b8ba03c9a323ddc9"}
Jul 17 21:58:28.281090 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:28.281005    2563 scope.go:117] "RemoveContainer" containerID="443ddb6c9cec6f04a7f838ec9a7bfe51841bc263f4b29d841122e182b03cac4f"
Jul 17 21:58:29.288103 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:29.288062    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerID="c80c40f8f081d135e6c4dfc4d1e3952ca6e86ed8d6640cba4b632f44d4ea5058" exitCode=6
Jul 17 21:58:29.288507 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:29.288148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" event={"ID":"c6af6f12-065d-4cb4-b06e-a773eeb85438","Type":"ContainerDied","Data":"c80c40f8f081d135e6c4dfc4d1e3952ca6e86ed8d6640cba4b632f44d4ea5058"}
Jul 17 21:58:29.288507 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:29.288240    2563 scope.go:117] "RemoveContainer" containerID="443ddb6c9cec6f04a7f838ec9a7bfe51841bc263f4b29d841122e182b03cac4f"
Jul 17 21:58:29.288507 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:29.288451    2563 scope.go:117] "RemoveContainer" containerID="c80c40f8f081d135e6c4dfc4d1e3952ca6e86ed8d6640cba4b632f44d4ea5058"
Jul 17 21:58:29.288743 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:29.288723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559_odh-ai-gateway-infra(c6af6f12-065d-4cb4-b06e-a773eeb85438)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438"
Jul 17 21:58:30.294719 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:30.294691    2563 scope.go:117] "RemoveContainer" containerID="c80c40f8f081d135e6c4dfc4d1e3952ca6e86ed8d6640cba4b632f44d4ea5058"
Jul 17 21:58:30.295077 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:30.294871    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559_odh-ai-gateway-infra(c6af6f12-065d-4cb4-b06e-a773eeb85438)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438"
Jul 17 21:58:35.596851 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:35.596814    2563 scope.go:117] "RemoveContainer" containerID="58e76ac2cb60614ff52208db0a6cd3dc988e227a2e52263ac1060df245d2a04c"
Jul 17 21:58:36.323873 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:36.323832    2563 generic.go:356] "Generic (PLEG): container finished" podID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerID="3b187b1817fa5354644e684d628c49af2cd043c569b5ca1b0765eae1a58b9872" exitCode=6
Jul 17 21:58:36.324088 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:36.323907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" event={"ID":"66e2db6d-0c32-40bd-84b7-e78fb6833306","Type":"ContainerDied","Data":"3b187b1817fa5354644e684d628c49af2cd043c569b5ca1b0765eae1a58b9872"}
Jul 17 21:58:36.324088 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:36.323993    2563 scope.go:117] "RemoveContainer" containerID="58e76ac2cb60614ff52208db0a6cd3dc988e227a2e52263ac1060df245d2a04c"
Jul 17 21:58:36.324365 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:36.324338    2563 scope.go:117] "RemoveContainer" containerID="3b187b1817fa5354644e684d628c49af2cd043c569b5ca1b0765eae1a58b9872"
Jul 17 21:58:36.324587 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:36.324567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-467f4_odh-ai-gateway-infra(66e2db6d-0c32-40bd-84b7-e78fb6833306)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306"
Jul 17 21:58:37.349829 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:37.349796    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"]
Jul 17 21:58:37.489878 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:37.489855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:37.591120 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:37.591084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e2db6d-0c32-40bd-84b7-e78fb6833306-maas-api-service-ca\") pod \"66e2db6d-0c32-40bd-84b7-e78fb6833306\" (UID: \"66e2db6d-0c32-40bd-84b7-e78fb6833306\") "
Jul 17 21:58:37.591482 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:37.591454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e2db6d-0c32-40bd-84b7-e78fb6833306-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66e2db6d-0c32-40bd-84b7-e78fb6833306" (UID: "66e2db6d-0c32-40bd-84b7-e78fb6833306"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:37.692393 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:37.692359    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e2db6d-0c32-40bd-84b7-e78fb6833306-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:58:38.336010 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:38.335983    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"
Jul 17 21:58:38.336178 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:38.335983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4" event={"ID":"66e2db6d-0c32-40bd-84b7-e78fb6833306","Type":"ContainerDied","Data":"5baf66222febc7aad8ddd0eed31adcc96733378d3929d0148c189053efceded4"}
Jul 17 21:58:38.336178 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:38.336125    2563 scope.go:117] "RemoveContainer" containerID="3b187b1817fa5354644e684d628c49af2cd043c569b5ca1b0765eae1a58b9872"
Jul 17 21:58:38.357533 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:38.357508    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"]
Jul 17 21:58:38.362942 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:38.362921    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-467f4"]
Jul 17 21:58:38.602751 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:38.602653    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" path="/var/lib/kubelet/pods/66e2db6d-0c32-40bd-84b7-e78fb6833306/volumes"
Jul 17 21:58:43.597523 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:43.597490    2563 scope.go:117] "RemoveContainer" containerID="c80c40f8f081d135e6c4dfc4d1e3952ca6e86ed8d6640cba4b632f44d4ea5058"
Jul 17 21:58:44.366507 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:44.366473    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerID="99da80c8f33b430b2934f483fbc81fbbed362b6c066c93a49d5dc0fd258b6f7b" exitCode=6
Jul 17 21:58:44.366699 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:44.366555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" event={"ID":"c6af6f12-065d-4cb4-b06e-a773eeb85438","Type":"ContainerDied","Data":"99da80c8f33b430b2934f483fbc81fbbed362b6c066c93a49d5dc0fd258b6f7b"}
Jul 17 21:58:44.366699 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:44.366646    2563 scope.go:117] "RemoveContainer" containerID="c80c40f8f081d135e6c4dfc4d1e3952ca6e86ed8d6640cba4b632f44d4ea5058"
Jul 17 21:58:44.366950 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:44.366931    2563 scope.go:117] "RemoveContainer" containerID="99da80c8f33b430b2934f483fbc81fbbed362b6c066c93a49d5dc0fd258b6f7b"
Jul 17 21:58:44.367162 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:58:44.367134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559_odh-ai-gateway-infra(c6af6f12-065d-4cb4-b06e-a773eeb85438)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438"
Jul 17 21:58:45.390859 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:45.390826    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"]
Jul 17 21:58:45.527057 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:45.527032    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:45.661307 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:45.661284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6af6f12-065d-4cb4-b06e-a773eeb85438-maas-api-service-ca\") pod \"c6af6f12-065d-4cb4-b06e-a773eeb85438\" (UID: \"c6af6f12-065d-4cb4-b06e-a773eeb85438\") "
Jul 17 21:58:45.661679 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:45.661659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6af6f12-065d-4cb4-b06e-a773eeb85438-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6af6f12-065d-4cb4-b06e-a773eeb85438" (UID: "c6af6f12-065d-4cb4-b06e-a773eeb85438"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:45.762003 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:45.761969    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6af6f12-065d-4cb4-b06e-a773eeb85438-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:58:46.378486 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:46.378458    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"
Jul 17 21:58:46.378720 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:46.378456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559" event={"ID":"c6af6f12-065d-4cb4-b06e-a773eeb85438","Type":"ContainerDied","Data":"2577935fd060bc78f4e6956e2cccecdccdeaaaf593b063a6b8ba03c9a323ddc9"}
Jul 17 21:58:46.378720 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:46.378635    2563 scope.go:117] "RemoveContainer" containerID="99da80c8f33b430b2934f483fbc81fbbed362b6c066c93a49d5dc0fd258b6f7b"
Jul 17 21:58:46.399362 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:46.399335    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"]
Jul 17 21:58:46.403190 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:46.403164    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-sn559"]
Jul 17 21:58:46.603083 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:58:46.603050    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" path="/var/lib/kubelet/pods/c6af6f12-065d-4cb4-b06e-a773eeb85438/volumes"
Jul 17 21:59:05.037461 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.037422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"]
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.037960    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.037983    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038003    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038022    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038031    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038041    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038073    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038082 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038080    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038103    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038111    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038209    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038220    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038229    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7922414b-5fa7-4aa8-b002-a945a3c85c36" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038238    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6af6f12-065d-4cb4-b06e-a773eeb85438" containerName="revoke-keys"
Jul 17 21:59:05.038752 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.038255    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e2db6d-0c32-40bd-84b7-e78fb6833306" containerName="revoke-keys"
Jul 17 21:59:05.041743 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.041718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:05.044015 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.043995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:59:05.048427 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.048398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"]
Jul 17 21:59:05.232898 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.232861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b55a645-570e-41c3-afba-025f596f9c8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn\" (UID: \"5b55a645-570e-41c3-afba-025f596f9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:05.334387 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.334297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b55a645-570e-41c3-afba-025f596f9c8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn\" (UID: \"5b55a645-570e-41c3-afba-025f596f9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:05.334905 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.334886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b55a645-570e-41c3-afba-025f596f9c8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn\" (UID: \"5b55a645-570e-41c3-afba-025f596f9c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:05.354304 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.354274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:05.491885 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:05.491861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"]
Jul 17 21:59:05.494331 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:59:05.494302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b55a645_570e_41c3_afba_025f596f9c8d.slice/crio-2eb0b5e116f882ef164b08db4c58fd0f6343d0ea277dc78819c46bfd57a31b61 WatchSource:0}: Error finding container 2eb0b5e116f882ef164b08db4c58fd0f6343d0ea277dc78819c46bfd57a31b61: Status 404 returned error can't find the container with id 2eb0b5e116f882ef164b08db4c58fd0f6343d0ea277dc78819c46bfd57a31b61
Jul 17 21:59:06.472820 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:06.472783    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b55a645-570e-41c3-afba-025f596f9c8d" containerID="835afa474ed5d95ee1eda3c1c0248fabb7f0ba8b4105ad203558582950c20cf8" exitCode=7
Jul 17 21:59:06.473228 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:06.472840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" event={"ID":"5b55a645-570e-41c3-afba-025f596f9c8d","Type":"ContainerDied","Data":"835afa474ed5d95ee1eda3c1c0248fabb7f0ba8b4105ad203558582950c20cf8"}
Jul 17 21:59:06.473228 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:06.472882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" event={"ID":"5b55a645-570e-41c3-afba-025f596f9c8d","Type":"ContainerStarted","Data":"2eb0b5e116f882ef164b08db4c58fd0f6343d0ea277dc78819c46bfd57a31b61"}
Jul 17 21:59:06.473228 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:06.473075    2563 scope.go:117] "RemoveContainer" containerID="835afa474ed5d95ee1eda3c1c0248fabb7f0ba8b4105ad203558582950c20cf8"
Jul 17 21:59:07.480004 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:07.479964    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b55a645-570e-41c3-afba-025f596f9c8d" containerID="9a2cff126a5a998c75ad5a2824b518093201a5254751c64e7e3a637b18104cca" exitCode=7
Jul 17 21:59:07.480426 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:07.480051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" event={"ID":"5b55a645-570e-41c3-afba-025f596f9c8d","Type":"ContainerDied","Data":"9a2cff126a5a998c75ad5a2824b518093201a5254751c64e7e3a637b18104cca"}
Jul 17 21:59:07.480426 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:07.480113    2563 scope.go:117] "RemoveContainer" containerID="835afa474ed5d95ee1eda3c1c0248fabb7f0ba8b4105ad203558582950c20cf8"
Jul 17 21:59:07.480426 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:07.480217    2563 scope.go:117] "RemoveContainer" containerID="9a2cff126a5a998c75ad5a2824b518093201a5254751c64e7e3a637b18104cca"
Jul 17 21:59:07.480556 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:07.480446    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn_odh-ai-gateway-infra(5b55a645-570e-41c3-afba-025f596f9c8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" podUID="5b55a645-570e-41c3-afba-025f596f9c8d"
Jul 17 21:59:08.487275 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:08.487245    2563 scope.go:117] "RemoveContainer" containerID="9a2cff126a5a998c75ad5a2824b518093201a5254751c64e7e3a637b18104cca"
Jul 17 21:59:08.487751 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:08.487478    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn_odh-ai-gateway-infra(5b55a645-570e-41c3-afba-025f596f9c8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" podUID="5b55a645-570e-41c3-afba-025f596f9c8d"
Jul 17 21:59:09.383136 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.383099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"]
Jul 17 21:59:09.387902 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.387885    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:09.395035 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.395000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"]
Jul 17 21:59:09.470518 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.470475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851c3d0c-aa13-496d-ad27-b5ccc28b2654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-djssm\" (UID: \"851c3d0c-aa13-496d-ad27-b5ccc28b2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:09.570998 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.570961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851c3d0c-aa13-496d-ad27-b5ccc28b2654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-djssm\" (UID: \"851c3d0c-aa13-496d-ad27-b5ccc28b2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:09.571588 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.571565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851c3d0c-aa13-496d-ad27-b5ccc28b2654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-djssm\" (UID: \"851c3d0c-aa13-496d-ad27-b5ccc28b2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:09.699850 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:09.699814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:10.041468 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:10.041445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"]
Jul 17 21:59:10.044105 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:59:10.044080    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851c3d0c_aa13_496d_ad27_b5ccc28b2654.slice/crio-f2e48e3366315bb772fb6afb0b574ee04097dc8a8e6603fdc3aa3928bc3d421e WatchSource:0}: Error finding container f2e48e3366315bb772fb6afb0b574ee04097dc8a8e6603fdc3aa3928bc3d421e: Status 404 returned error can't find the container with id f2e48e3366315bb772fb6afb0b574ee04097dc8a8e6603fdc3aa3928bc3d421e
Jul 17 21:59:10.497376 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:10.497338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" event={"ID":"851c3d0c-aa13-496d-ad27-b5ccc28b2654","Type":"ContainerStarted","Data":"e533ef0e5d835be779b2b3ffd30bca28d6ba202f1608f7344517c98c1cba3ecc"}
Jul 17 21:59:10.497581 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:10.497386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" event={"ID":"851c3d0c-aa13-496d-ad27-b5ccc28b2654","Type":"ContainerStarted","Data":"f2e48e3366315bb772fb6afb0b574ee04097dc8a8e6603fdc3aa3928bc3d421e"}
Jul 17 21:59:10.513307 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:10.513257    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" podStartSLOduration=1.513241602 podStartE2EDuration="1.513241602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:10.511927653 +0000 UTC m=+1914.424167941" watchObservedRunningTime="2026-07-17 21:59:10.513241602 +0000 UTC m=+1914.425481891"
Jul 17 21:59:11.503711 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:11.503674    2563 generic.go:356] "Generic (PLEG): container finished" podID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerID="e533ef0e5d835be779b2b3ffd30bca28d6ba202f1608f7344517c98c1cba3ecc" exitCode=6
Jul 17 21:59:11.504107 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:11.503735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" event={"ID":"851c3d0c-aa13-496d-ad27-b5ccc28b2654","Type":"ContainerDied","Data":"e533ef0e5d835be779b2b3ffd30bca28d6ba202f1608f7344517c98c1cba3ecc"}
Jul 17 21:59:11.504107 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:11.503958    2563 scope.go:117] "RemoveContainer" containerID="e533ef0e5d835be779b2b3ffd30bca28d6ba202f1608f7344517c98c1cba3ecc"
Jul 17 21:59:12.510500 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:12.510470    2563 generic.go:356] "Generic (PLEG): container finished" podID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerID="698aee3ac02650e96611961a56cd49549ed0bf9028f53d2fc2fa4d7b988918bc" exitCode=6
Jul 17 21:59:12.510930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:12.510509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" event={"ID":"851c3d0c-aa13-496d-ad27-b5ccc28b2654","Type":"ContainerDied","Data":"698aee3ac02650e96611961a56cd49549ed0bf9028f53d2fc2fa4d7b988918bc"}
Jul 17 21:59:12.510930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:12.510573    2563 scope.go:117] "RemoveContainer" containerID="e533ef0e5d835be779b2b3ffd30bca28d6ba202f1608f7344517c98c1cba3ecc"
Jul 17 21:59:12.510930 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:12.510765    2563 scope.go:117] "RemoveContainer" containerID="698aee3ac02650e96611961a56cd49549ed0bf9028f53d2fc2fa4d7b988918bc"
Jul 17 21:59:12.511105 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:12.510975    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-djssm_odh-ai-gateway-infra(851c3d0c-aa13-496d-ad27-b5ccc28b2654)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654"
Jul 17 21:59:13.516861 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:13.516830    2563 scope.go:117] "RemoveContainer" containerID="698aee3ac02650e96611961a56cd49549ed0bf9028f53d2fc2fa4d7b988918bc"
Jul 17 21:59:13.517254 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:13.517020    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-djssm_odh-ai-gateway-infra(851c3d0c-aa13-496d-ad27-b5ccc28b2654)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654"
Jul 17 21:59:17.172718 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.172680    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"]
Jul 17 21:59:17.177887 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.177869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:17.183582 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.183555    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"]
Jul 17 21:59:17.235912 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.235873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80f924-d601-40d5-a683-b8796989e470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82\" (UID: \"af80f924-d601-40d5-a683-b8796989e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:17.337249 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.337209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80f924-d601-40d5-a683-b8796989e470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82\" (UID: \"af80f924-d601-40d5-a683-b8796989e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:17.337918 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.337897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80f924-d601-40d5-a683-b8796989e470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82\" (UID: \"af80f924-d601-40d5-a683-b8796989e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:17.489324 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.489238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:17.625638 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:17.625610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"]
Jul 17 21:59:17.627972 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:59:17.627937    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf80f924_d601_40d5_a683_b8796989e470.slice/crio-feb370f515d4e14f7d40d02165851d917f32e71a2d67b8a8f75534163cb4ec7b WatchSource:0}: Error finding container feb370f515d4e14f7d40d02165851d917f32e71a2d67b8a8f75534163cb4ec7b: Status 404 returned error can't find the container with id feb370f515d4e14f7d40d02165851d917f32e71a2d67b8a8f75534163cb4ec7b
Jul 17 21:59:18.551987 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:18.551894    2563 generic.go:356] "Generic (PLEG): container finished" podID="af80f924-d601-40d5-a683-b8796989e470" containerID="d05065698cc2cdd2d33e04c20cd2db09abed7554d2c239efa59b14f1bd57e090" exitCode=6
Jul 17 21:59:18.552375 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:18.551978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" event={"ID":"af80f924-d601-40d5-a683-b8796989e470","Type":"ContainerDied","Data":"d05065698cc2cdd2d33e04c20cd2db09abed7554d2c239efa59b14f1bd57e090"}
Jul 17 21:59:18.552375 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:18.552053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" event={"ID":"af80f924-d601-40d5-a683-b8796989e470","Type":"ContainerStarted","Data":"feb370f515d4e14f7d40d02165851d917f32e71a2d67b8a8f75534163cb4ec7b"}
Jul 17 21:59:18.552375 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:18.552183    2563 scope.go:117] "RemoveContainer" containerID="d05065698cc2cdd2d33e04c20cd2db09abed7554d2c239efa59b14f1bd57e090"
Jul 17 21:59:19.558517 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:19.558481    2563 generic.go:356] "Generic (PLEG): container finished" podID="af80f924-d601-40d5-a683-b8796989e470" containerID="e40980ab1a29315779b0c9b28b8d708744c00ff8dac5cd3b7669721c543397e7" exitCode=6
Jul 17 21:59:19.558924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:19.558534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" event={"ID":"af80f924-d601-40d5-a683-b8796989e470","Type":"ContainerDied","Data":"e40980ab1a29315779b0c9b28b8d708744c00ff8dac5cd3b7669721c543397e7"}
Jul 17 21:59:19.558924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:19.558592    2563 scope.go:117] "RemoveContainer" containerID="d05065698cc2cdd2d33e04c20cd2db09abed7554d2c239efa59b14f1bd57e090"
Jul 17 21:59:19.558924 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:19.558836    2563 scope.go:117] "RemoveContainer" containerID="e40980ab1a29315779b0c9b28b8d708744c00ff8dac5cd3b7669721c543397e7"
Jul 17 21:59:19.559072 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:19.559029    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82_odh-ai-gateway-infra(af80f924-d601-40d5-a683-b8796989e470)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" podUID="af80f924-d601-40d5-a683-b8796989e470"
Jul 17 21:59:19.596469 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:19.596445    2563 scope.go:117] "RemoveContainer" containerID="9a2cff126a5a998c75ad5a2824b518093201a5254751c64e7e3a637b18104cca"
Jul 17 21:59:20.568231 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:20.568140    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b55a645-570e-41c3-afba-025f596f9c8d" containerID="ece7f500513964507d01993dfe50d7f74e3347d8eb10201c4715c3da3d573b04" exitCode=7
Jul 17 21:59:20.568231 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:20.568220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" event={"ID":"5b55a645-570e-41c3-afba-025f596f9c8d","Type":"ContainerDied","Data":"ece7f500513964507d01993dfe50d7f74e3347d8eb10201c4715c3da3d573b04"}
Jul 17 21:59:20.568792 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:20.568277    2563 scope.go:117] "RemoveContainer" containerID="9a2cff126a5a998c75ad5a2824b518093201a5254751c64e7e3a637b18104cca"
Jul 17 21:59:20.568792 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:20.568650    2563 scope.go:117] "RemoveContainer" containerID="ece7f500513964507d01993dfe50d7f74e3347d8eb10201c4715c3da3d573b04"
Jul 17 21:59:20.568900 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:20.568856    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn_odh-ai-gateway-infra(5b55a645-570e-41c3-afba-025f596f9c8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" podUID="5b55a645-570e-41c3-afba-025f596f9c8d"
Jul 17 21:59:20.570170 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:20.570153    2563 scope.go:117] "RemoveContainer" containerID="e40980ab1a29315779b0c9b28b8d708744c00ff8dac5cd3b7669721c543397e7"
Jul 17 21:59:20.570299 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:20.570283    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82_odh-ai-gateway-infra(af80f924-d601-40d5-a683-b8796989e470)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" podUID="af80f924-d601-40d5-a683-b8796989e470"
Jul 17 21:59:21.595398 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:21.595368    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"]
Jul 17 21:59:21.736206 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:21.736173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:21.777100 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:21.777063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b55a645-570e-41c3-afba-025f596f9c8d-maas-api-service-ca\") pod \"5b55a645-570e-41c3-afba-025f596f9c8d\" (UID: \"5b55a645-570e-41c3-afba-025f596f9c8d\") "
Jul 17 21:59:21.777409 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:21.777382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b55a645-570e-41c3-afba-025f596f9c8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b55a645-570e-41c3-afba-025f596f9c8d" (UID: "5b55a645-570e-41c3-afba-025f596f9c8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:21.878653 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:21.878562    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b55a645-570e-41c3-afba-025f596f9c8d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.582519 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:22.582491    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"
Jul 17 21:59:22.582719 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:22.582494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn" event={"ID":"5b55a645-570e-41c3-afba-025f596f9c8d","Type":"ContainerDied","Data":"2eb0b5e116f882ef164b08db4c58fd0f6343d0ea277dc78819c46bfd57a31b61"}
Jul 17 21:59:22.582719 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:22.582659    2563 scope.go:117] "RemoveContainer" containerID="ece7f500513964507d01993dfe50d7f74e3347d8eb10201c4715c3da3d573b04"
Jul 17 21:59:22.621166 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:22.621134    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"]
Jul 17 21:59:22.623060 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:22.623036    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tnvpn"]
Jul 17 21:59:24.602181 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:24.602146    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" path="/var/lib/kubelet/pods/5b55a645-570e-41c3-afba-025f596f9c8d/volumes"
Jul 17 21:59:27.596652 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:27.596593    2563 scope.go:117] "RemoveContainer" containerID="698aee3ac02650e96611961a56cd49549ed0bf9028f53d2fc2fa4d7b988918bc"
Jul 17 21:59:28.613814 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:28.613783    2563 generic.go:356] "Generic (PLEG): container finished" podID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerID="21ec6f16b087ab5bc5e0ff24a98b869f22c3db749bc53c58a37fe3fe7ccc28d1" exitCode=6
Jul 17 21:59:28.614299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:28.613854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" event={"ID":"851c3d0c-aa13-496d-ad27-b5ccc28b2654","Type":"ContainerDied","Data":"21ec6f16b087ab5bc5e0ff24a98b869f22c3db749bc53c58a37fe3fe7ccc28d1"}
Jul 17 21:59:28.614299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:28.613909    2563 scope.go:117] "RemoveContainer" containerID="698aee3ac02650e96611961a56cd49549ed0bf9028f53d2fc2fa4d7b988918bc"
Jul 17 21:59:28.614299 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:28.614224    2563 scope.go:117] "RemoveContainer" containerID="21ec6f16b087ab5bc5e0ff24a98b869f22c3db749bc53c58a37fe3fe7ccc28d1"
Jul 17 21:59:28.614493 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:28.614469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-djssm_odh-ai-gateway-infra(851c3d0c-aa13-496d-ad27-b5ccc28b2654)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654"
Jul 17 21:59:29.638141 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:29.638108    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"]
Jul 17 21:59:29.785063 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:29.785036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:29.851701 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:29.851670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851c3d0c-aa13-496d-ad27-b5ccc28b2654-maas-api-service-ca\") pod \"851c3d0c-aa13-496d-ad27-b5ccc28b2654\" (UID: \"851c3d0c-aa13-496d-ad27-b5ccc28b2654\") "
Jul 17 21:59:29.852067 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:29.852039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851c3d0c-aa13-496d-ad27-b5ccc28b2654-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "851c3d0c-aa13-496d-ad27-b5ccc28b2654" (UID: "851c3d0c-aa13-496d-ad27-b5ccc28b2654"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:29.952640 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:29.952610    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851c3d0c-aa13-496d-ad27-b5ccc28b2654-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:59:30.625988 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:30.625962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"
Jul 17 21:59:30.626150 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:30.625996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm" event={"ID":"851c3d0c-aa13-496d-ad27-b5ccc28b2654","Type":"ContainerDied","Data":"f2e48e3366315bb772fb6afb0b574ee04097dc8a8e6603fdc3aa3928bc3d421e"}
Jul 17 21:59:30.626150 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:30.626058    2563 scope.go:117] "RemoveContainer" containerID="21ec6f16b087ab5bc5e0ff24a98b869f22c3db749bc53c58a37fe3fe7ccc28d1"
Jul 17 21:59:30.640755 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:30.640729    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"]
Jul 17 21:59:30.647733 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:30.647713    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-djssm"]
Jul 17 21:59:32.602606 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:32.602561    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" path="/var/lib/kubelet/pods/851c3d0c-aa13-496d-ad27-b5ccc28b2654/volumes"
Jul 17 21:59:35.596687 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:35.596657    2563 scope.go:117] "RemoveContainer" containerID="e40980ab1a29315779b0c9b28b8d708744c00ff8dac5cd3b7669721c543397e7"
Jul 17 21:59:36.654669 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:36.654630    2563 generic.go:356] "Generic (PLEG): container finished" podID="af80f924-d601-40d5-a683-b8796989e470" containerID="f36a18f814f0a7e05f1c3e7dc5dec269ea4887e157e6226c18aedb18c1358977" exitCode=6
Jul 17 21:59:36.654669 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:36.654675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" event={"ID":"af80f924-d601-40d5-a683-b8796989e470","Type":"ContainerDied","Data":"f36a18f814f0a7e05f1c3e7dc5dec269ea4887e157e6226c18aedb18c1358977"}
Jul 17 21:59:36.655221 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:36.654723    2563 scope.go:117] "RemoveContainer" containerID="e40980ab1a29315779b0c9b28b8d708744c00ff8dac5cd3b7669721c543397e7"
Jul 17 21:59:36.655221 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:36.655030    2563 scope.go:117] "RemoveContainer" containerID="f36a18f814f0a7e05f1c3e7dc5dec269ea4887e157e6226c18aedb18c1358977"
Jul 17 21:59:36.655335 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:36.655279    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82_odh-ai-gateway-infra(af80f924-d601-40d5-a683-b8796989e470)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" podUID="af80f924-d601-40d5-a683-b8796989e470"
Jul 17 21:59:37.679043 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:37.679012    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"]
Jul 17 21:59:37.820236 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:37.820212    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:37.921904 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:37.921856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80f924-d601-40d5-a683-b8796989e470-maas-api-service-ca\") pod \"af80f924-d601-40d5-a683-b8796989e470\" (UID: \"af80f924-d601-40d5-a683-b8796989e470\") "
Jul 17 21:59:37.922214 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:37.922190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af80f924-d601-40d5-a683-b8796989e470-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af80f924-d601-40d5-a683-b8796989e470" (UID: "af80f924-d601-40d5-a683-b8796989e470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:38.023380 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:38.023278    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80f924-d601-40d5-a683-b8796989e470-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 21:59:38.665555 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:38.665523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"
Jul 17 21:59:38.665747 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:38.665556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82" event={"ID":"af80f924-d601-40d5-a683-b8796989e470","Type":"ContainerDied","Data":"feb370f515d4e14f7d40d02165851d917f32e71a2d67b8a8f75534163cb4ec7b"}
Jul 17 21:59:38.665747 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:38.665632    2563 scope.go:117] "RemoveContainer" containerID="f36a18f814f0a7e05f1c3e7dc5dec269ea4887e157e6226c18aedb18c1358977"
Jul 17 21:59:38.682734 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:38.682710    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"]
Jul 17 21:59:38.690313 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:38.690290    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4dn82"]
Jul 17 21:59:40.602325 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:40.602290    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af80f924-d601-40d5-a683-b8796989e470" path="/var/lib/kubelet/pods/af80f924-d601-40d5-a683-b8796989e470/volumes"
Jul 17 21:59:55.077491 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"]
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077853    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077878    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077891    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077896    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077907    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077917    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077921    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.077965 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.077993    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078008    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078014    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078020    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078025    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078030    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.078356 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.078036    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 21:59:55.080971 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.080954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 21:59:55.083246 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.083222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 21:59:55.087665 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.087514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"]
Jul 17 21:59:55.181933 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.181899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee61292d-4247-4cf8-a158-3eb1abd5778d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg\" (UID: \"ee61292d-4247-4cf8-a158-3eb1abd5778d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 21:59:55.283252 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.283210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee61292d-4247-4cf8-a158-3eb1abd5778d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg\" (UID: \"ee61292d-4247-4cf8-a158-3eb1abd5778d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 21:59:55.283881 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.283860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee61292d-4247-4cf8-a158-3eb1abd5778d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg\" (UID: \"ee61292d-4247-4cf8-a158-3eb1abd5778d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 21:59:55.393631 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.393526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 21:59:55.737088 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.737061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"]
Jul 17 21:59:55.739526 ip-10-0-139-16 kubenswrapper[2563]: W0717 21:59:55.739494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee61292d_4247_4cf8_a158_3eb1abd5778d.slice/crio-f00dd3a1b0d707bfca7caff388f9a322a13130571c6b29658600af68eeee4eb2 WatchSource:0}: Error finding container f00dd3a1b0d707bfca7caff388f9a322a13130571c6b29658600af68eeee4eb2: Status 404 returned error can't find the container with id f00dd3a1b0d707bfca7caff388f9a322a13130571c6b29658600af68eeee4eb2
Jul 17 21:59:55.749580 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:55.749549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" event={"ID":"ee61292d-4247-4cf8-a158-3eb1abd5778d","Type":"ContainerStarted","Data":"f00dd3a1b0d707bfca7caff388f9a322a13130571c6b29658600af68eeee4eb2"}
Jul 17 21:59:56.755514 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:56.755424    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerID="5d11564bd12a455f94bcc515dae4cf4516888ed075bfcff9d0f21c089ed2d687" exitCode=7
Jul 17 21:59:56.755904 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:56.755512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" event={"ID":"ee61292d-4247-4cf8-a158-3eb1abd5778d","Type":"ContainerDied","Data":"5d11564bd12a455f94bcc515dae4cf4516888ed075bfcff9d0f21c089ed2d687"}
Jul 17 21:59:56.755904 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:56.755729    2563 scope.go:117] "RemoveContainer" containerID="5d11564bd12a455f94bcc515dae4cf4516888ed075bfcff9d0f21c089ed2d687"
Jul 17 21:59:57.762391 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:57.762360    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerID="86e6fb3f1b83e80ee54362e639dd1b0364f9316ae836f142d0dd6f01b853acda" exitCode=7
Jul 17 21:59:57.762894 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:57.762405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" event={"ID":"ee61292d-4247-4cf8-a158-3eb1abd5778d","Type":"ContainerDied","Data":"86e6fb3f1b83e80ee54362e639dd1b0364f9316ae836f142d0dd6f01b853acda"}
Jul 17 21:59:57.762894 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:57.762452    2563 scope.go:117] "RemoveContainer" containerID="5d11564bd12a455f94bcc515dae4cf4516888ed075bfcff9d0f21c089ed2d687"
Jul 17 21:59:57.762894 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:57.762679    2563 scope.go:117] "RemoveContainer" containerID="86e6fb3f1b83e80ee54362e639dd1b0364f9316ae836f142d0dd6f01b853acda"
Jul 17 21:59:57.763063 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:57.762945    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg_odh-ai-gateway-infra(ee61292d-4247-4cf8-a158-3eb1abd5778d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d"
Jul 17 21:59:58.769331 ip-10-0-139-16 kubenswrapper[2563]: I0717 21:59:58.769301    2563 scope.go:117] "RemoveContainer" containerID="86e6fb3f1b83e80ee54362e639dd1b0364f9316ae836f142d0dd6f01b853acda"
Jul 17 21:59:58.769728 ip-10-0-139-16 kubenswrapper[2563]: E0717 21:59:58.769510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg_odh-ai-gateway-infra(ee61292d-4247-4cf8-a158-3eb1abd5778d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d"
Jul 17 22:00:00.143661 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.143627    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"]
Jul 17 22:00:00.144011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.143986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 22:00:00.144011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.143996    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="851c3d0c-aa13-496d-ad27-b5ccc28b2654" containerName="revoke-keys"
Jul 17 22:00:00.144138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.144014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 22:00:00.144138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.144020    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af80f924-d601-40d5-a683-b8796989e470" containerName="revoke-keys"
Jul 17 22:00:00.144138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.144029    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 22:00:00.144138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.144034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 22:00:00.144138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.144115    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b55a645-570e-41c3-afba-025f596f9c8d" containerName="revoke-keys"
Jul 17 22:00:00.154001 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.153958    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"]
Jul 17 22:00:00.154197 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.154089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"
Jul 17 22:00:00.159815 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.159645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"]
Jul 17 22:00:00.159815 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.159688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"]
Jul 17 22:00:00.159815 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.159770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"
Jul 17 22:00:00.172216 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.172187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"
Jul 17 22:00:00.180459 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.180430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"
Jul 17 22:00:00.528329 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.528285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"]
Jul 17 22:00:00.529907 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:00:00.529876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90a159c_47e1_4d36_a9ea_017373f65754.slice/crio-af3c80d3e0e4e12075dcfe1d2884161da9aef7cfdf3e35a6f286e6a7ae2f0e8b WatchSource:0}: Error finding container af3c80d3e0e4e12075dcfe1d2884161da9aef7cfdf3e35a6f286e6a7ae2f0e8b: Status 404 returned error can't find the container with id af3c80d3e0e4e12075dcfe1d2884161da9aef7cfdf3e35a6f286e6a7ae2f0e8b
Jul 17 22:00:00.550667 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.550643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"]
Jul 17 22:00:00.553111 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:00:00.553086    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669e1750_d049_4193_819d_50b724a38806.slice/crio-20658058001b68896e4687e92cc2e5f4ff6b700594f07cbeaf7e0e0ec447df0f WatchSource:0}: Error finding container 20658058001b68896e4687e92cc2e5f4ff6b700594f07cbeaf7e0e0ec447df0f: Status 404 returned error can't find the container with id 20658058001b68896e4687e92cc2e5f4ff6b700594f07cbeaf7e0e0ec447df0f
Jul 17 22:00:00.780382 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.780289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw" event={"ID":"e90a159c-47e1-4d36-a9ea-017373f65754","Type":"ContainerStarted","Data":"af3c80d3e0e4e12075dcfe1d2884161da9aef7cfdf3e35a6f286e6a7ae2f0e8b"}
Jul 17 22:00:00.781413 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:00.781389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" event={"ID":"669e1750-d049-4193-819d-50b724a38806","Type":"ContainerStarted","Data":"20658058001b68896e4687e92cc2e5f4ff6b700594f07cbeaf7e0e0ec447df0f"}
Jul 17 22:00:01.787384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:01.787350    2563 generic.go:356] "Generic (PLEG): container finished" podID="669e1750-d049-4193-819d-50b724a38806" containerID="d94077762e56b72c5028f8a265b12ec9bc5faa686c88c6cf1c6983e8229c94be" exitCode=7
Jul 17 22:00:01.787795 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:01.787434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" event={"ID":"669e1750-d049-4193-819d-50b724a38806","Type":"ContainerDied","Data":"d94077762e56b72c5028f8a265b12ec9bc5faa686c88c6cf1c6983e8229c94be"}
Jul 17 22:00:01.787795 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:01.787567    2563 scope.go:117] "RemoveContainer" containerID="d94077762e56b72c5028f8a265b12ec9bc5faa686c88c6cf1c6983e8229c94be"
Jul 17 22:00:01.789127 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:01.789103    2563 generic.go:356] "Generic (PLEG): container finished" podID="e90a159c-47e1-4d36-a9ea-017373f65754" containerID="d5cd5f2fd05458281bda904e67881e6ca0352416d4b0858bfe308d76102ea6fd" exitCode=0
Jul 17 22:00:01.789214 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:01.789157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw" event={"ID":"e90a159c-47e1-4d36-a9ea-017373f65754","Type":"ContainerDied","Data":"d5cd5f2fd05458281bda904e67881e6ca0352416d4b0858bfe308d76102ea6fd"}
Jul 17 22:00:02.796257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:02.796150    2563 generic.go:356] "Generic (PLEG): container finished" podID="669e1750-d049-4193-819d-50b724a38806" containerID="b980f67e9a75d358633d9845e1557eeb14815520174226133b4e5fb8590c715f" exitCode=7
Jul 17 22:00:02.796257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:02.796236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" event={"ID":"669e1750-d049-4193-819d-50b724a38806","Type":"ContainerDied","Data":"b980f67e9a75d358633d9845e1557eeb14815520174226133b4e5fb8590c715f"}
Jul 17 22:00:02.796771 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:02.796295    2563 scope.go:117] "RemoveContainer" containerID="d94077762e56b72c5028f8a265b12ec9bc5faa686c88c6cf1c6983e8229c94be"
Jul 17 22:00:02.796771 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:02.796473    2563 scope.go:117] "RemoveContainer" containerID="b980f67e9a75d358633d9845e1557eeb14815520174226133b4e5fb8590c715f"
Jul 17 22:00:02.796771 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:02.796738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk_odh-ai-gateway-infra(669e1750-d049-4193-819d-50b724a38806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" podUID="669e1750-d049-4193-819d-50b724a38806"
Jul 17 22:00:02.939265 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:02.939242    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"
Jul 17 22:00:03.802926 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:03.802900    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw"
Jul 17 22:00:03.802926 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:03.802910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-v2fmw" event={"ID":"e90a159c-47e1-4d36-a9ea-017373f65754","Type":"ContainerDied","Data":"af3c80d3e0e4e12075dcfe1d2884161da9aef7cfdf3e35a6f286e6a7ae2f0e8b"}
Jul 17 22:00:03.803407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:03.802961    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3c80d3e0e4e12075dcfe1d2884161da9aef7cfdf3e35a6f286e6a7ae2f0e8b"
Jul 17 22:00:03.804709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:03.804687    2563 scope.go:117] "RemoveContainer" containerID="b980f67e9a75d358633d9845e1557eeb14815520174226133b4e5fb8590c715f"
Jul 17 22:00:03.804945 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:03.804923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk_odh-ai-gateway-infra(669e1750-d049-4193-819d-50b724a38806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" podUID="669e1750-d049-4193-819d-50b724a38806"
Jul 17 22:00:09.426242 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.426158    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"]
Jul 17 22:00:09.426778 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.426618    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90a159c-47e1-4d36-a9ea-017373f65754" containerName="cleanup"
Jul 17 22:00:09.426778 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.426636    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90a159c-47e1-4d36-a9ea-017373f65754" containerName="cleanup"
Jul 17 22:00:09.426778 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.426716    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90a159c-47e1-4d36-a9ea-017373f65754" containerName="cleanup"
Jul 17 22:00:09.429812 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.429796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:09.437461 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.437441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"]
Jul 17 22:00:09.504741 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.504707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a230b4ac-dcdc-4cc2-a633-507bf2358f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w\" (UID: \"a230b4ac-dcdc-4cc2-a633-507bf2358f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:09.605928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.605895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a230b4ac-dcdc-4cc2-a633-507bf2358f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w\" (UID: \"a230b4ac-dcdc-4cc2-a633-507bf2358f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:09.606528 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.606506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a230b4ac-dcdc-4cc2-a633-507bf2358f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w\" (UID: \"a230b4ac-dcdc-4cc2-a633-507bf2358f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:09.742299 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.742218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:09.884627 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:09.884582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"]
Jul 17 22:00:09.886460 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:00:09.886432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda230b4ac_dcdc_4cc2_a633_507bf2358f9a.slice/crio-16f1a7ba55d09751dbe933622c0b56279ab34cdb1b2c397c06af4adb578b92ec WatchSource:0}: Error finding container 16f1a7ba55d09751dbe933622c0b56279ab34cdb1b2c397c06af4adb578b92ec: Status 404 returned error can't find the container with id 16f1a7ba55d09751dbe933622c0b56279ab34cdb1b2c397c06af4adb578b92ec
Jul 17 22:00:10.841878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:10.841787    2563 generic.go:356] "Generic (PLEG): container finished" podID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerID="d6c628cf93dfc15945c6646c11143f17cb553ed276fa704ffc560c8b3c290200" exitCode=6
Jul 17 22:00:10.842271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:10.841876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" event={"ID":"a230b4ac-dcdc-4cc2-a633-507bf2358f9a","Type":"ContainerDied","Data":"d6c628cf93dfc15945c6646c11143f17cb553ed276fa704ffc560c8b3c290200"}
Jul 17 22:00:10.842271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:10.841937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" event={"ID":"a230b4ac-dcdc-4cc2-a633-507bf2358f9a","Type":"ContainerStarted","Data":"16f1a7ba55d09751dbe933622c0b56279ab34cdb1b2c397c06af4adb578b92ec"}
Jul 17 22:00:10.842271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:10.842051    2563 scope.go:117] "RemoveContainer" containerID="d6c628cf93dfc15945c6646c11143f17cb553ed276fa704ffc560c8b3c290200"
Jul 17 22:00:11.848648 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:11.848549    2563 generic.go:356] "Generic (PLEG): container finished" podID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerID="0f46257f3f1f74db1731fbeb8b430ed10b270943482b42ca09cee5c8c06705cf" exitCode=6
Jul 17 22:00:11.849084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:11.848643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" event={"ID":"a230b4ac-dcdc-4cc2-a633-507bf2358f9a","Type":"ContainerDied","Data":"0f46257f3f1f74db1731fbeb8b430ed10b270943482b42ca09cee5c8c06705cf"}
Jul 17 22:00:11.849084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:11.848712    2563 scope.go:117] "RemoveContainer" containerID="d6c628cf93dfc15945c6646c11143f17cb553ed276fa704ffc560c8b3c290200"
Jul 17 22:00:11.849084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:11.848859    2563 scope.go:117] "RemoveContainer" containerID="0f46257f3f1f74db1731fbeb8b430ed10b270943482b42ca09cee5c8c06705cf"
Jul 17 22:00:11.849224 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:11.849109    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w_odh-ai-gateway-infra(a230b4ac-dcdc-4cc2-a633-507bf2358f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a"
Jul 17 22:00:12.856155 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:12.856132    2563 scope.go:117] "RemoveContainer" containerID="0f46257f3f1f74db1731fbeb8b430ed10b270943482b42ca09cee5c8c06705cf"
Jul 17 22:00:12.856537 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:12.856322    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w_odh-ai-gateway-infra(a230b4ac-dcdc-4cc2-a633-507bf2358f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a"
Jul 17 22:00:14.597024 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:14.596991    2563 scope.go:117] "RemoveContainer" containerID="86e6fb3f1b83e80ee54362e639dd1b0364f9316ae836f142d0dd6f01b853acda"
Jul 17 22:00:14.597439 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:14.597072    2563 scope.go:117] "RemoveContainer" containerID="b980f67e9a75d358633d9845e1557eeb14815520174226133b4e5fb8590c715f"
Jul 17 22:00:15.880030 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.879996    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerID="99dd80542a731142f05ea8be13e9da58046e1cf03fb1d0a36994b8f28851852a" exitCode=7
Jul 17 22:00:15.880480 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.880072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" event={"ID":"ee61292d-4247-4cf8-a158-3eb1abd5778d","Type":"ContainerDied","Data":"99dd80542a731142f05ea8be13e9da58046e1cf03fb1d0a36994b8f28851852a"}
Jul 17 22:00:15.880480 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.880184    2563 scope.go:117] "RemoveContainer" containerID="86e6fb3f1b83e80ee54362e639dd1b0364f9316ae836f142d0dd6f01b853acda"
Jul 17 22:00:15.880573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.880481    2563 scope.go:117] "RemoveContainer" containerID="99dd80542a731142f05ea8be13e9da58046e1cf03fb1d0a36994b8f28851852a"
Jul 17 22:00:15.880733 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:15.880712    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg_odh-ai-gateway-infra(ee61292d-4247-4cf8-a158-3eb1abd5778d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d"
Jul 17 22:00:15.882032 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.882013    2563 generic.go:356] "Generic (PLEG): container finished" podID="669e1750-d049-4193-819d-50b724a38806" containerID="f3100f7cbf4baf6bdd77f5855930f3bb5d03fe074ff1baa416bfab144d6e9449" exitCode=7
Jul 17 22:00:15.882124 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.882072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" event={"ID":"669e1750-d049-4193-819d-50b724a38806","Type":"ContainerDied","Data":"f3100f7cbf4baf6bdd77f5855930f3bb5d03fe074ff1baa416bfab144d6e9449"}
Jul 17 22:00:15.882289 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.882277    2563 scope.go:117] "RemoveContainer" containerID="f3100f7cbf4baf6bdd77f5855930f3bb5d03fe074ff1baa416bfab144d6e9449"
Jul 17 22:00:15.882451 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:15.882435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk_odh-ai-gateway-infra(669e1750-d049-4193-819d-50b724a38806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" podUID="669e1750-d049-4193-819d-50b724a38806"
Jul 17 22:00:15.891930 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:15.891904    2563 scope.go:117] "RemoveContainer" containerID="b980f67e9a75d358633d9845e1557eeb14815520174226133b4e5fb8590c715f"
Jul 17 22:00:16.904782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:16.904747    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"]
Jul 17 22:00:16.921177 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:16.921142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"]
Jul 17 22:00:17.072178 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.072151    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 22:00:17.106646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.106624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"
Jul 17 22:00:17.178118 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.178083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee61292d-4247-4cf8-a158-3eb1abd5778d-maas-api-service-ca\") pod \"ee61292d-4247-4cf8-a158-3eb1abd5778d\" (UID: \"ee61292d-4247-4cf8-a158-3eb1abd5778d\") "
Jul 17 22:00:17.178439 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.178416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee61292d-4247-4cf8-a158-3eb1abd5778d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee61292d-4247-4cf8-a158-3eb1abd5778d" (UID: "ee61292d-4247-4cf8-a158-3eb1abd5778d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:17.221112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221074    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"]
Jul 17 22:00:17.221478 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221465    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.221527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.221527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221501    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221507    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221520    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221533    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221623    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221630    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221648    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:17.221709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.221665    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:17.224746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.224729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:17.232391 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.232262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"]
Jul 17 22:00:17.279145 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.279108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e92d59-d21d-4303-80cb-1f52a12f215b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25\" (UID: \"10e92d59-d21d-4303-80cb-1f52a12f215b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:17.279330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.279226    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee61292d-4247-4cf8-a158-3eb1abd5778d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:00:17.380543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.380449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e92d59-d21d-4303-80cb-1f52a12f215b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25\" (UID: \"10e92d59-d21d-4303-80cb-1f52a12f215b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:17.381107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.381087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e92d59-d21d-4303-80cb-1f52a12f215b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25\" (UID: \"10e92d59-d21d-4303-80cb-1f52a12f215b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:17.537708 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.537669    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:17.676738 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.676713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"]
Jul 17 22:00:17.678366 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:00:17.678339    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e92d59_d21d_4303_80cb_1f52a12f215b.slice/crio-a8f98c323cb44b64ff3d53fd335a99fbb361d97661c9f9630fac0e48f224d46e WatchSource:0}: Error finding container a8f98c323cb44b64ff3d53fd335a99fbb361d97661c9f9630fac0e48f224d46e: Status 404 returned error can't find the container with id a8f98c323cb44b64ff3d53fd335a99fbb361d97661c9f9630fac0e48f224d46e
Jul 17 22:00:17.895659 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.895629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg" event={"ID":"ee61292d-4247-4cf8-a158-3eb1abd5778d","Type":"ContainerDied","Data":"f00dd3a1b0d707bfca7caff388f9a322a13130571c6b29658600af68eeee4eb2"}
Jul 17 22:00:17.895762 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.895694    2563 scope.go:117] "RemoveContainer" containerID="99dd80542a731142f05ea8be13e9da58046e1cf03fb1d0a36994b8f28851852a"
Jul 17 22:00:17.895762 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.895645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"
Jul 17 22:00:17.897388 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.897363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" event={"ID":"10e92d59-d21d-4303-80cb-1f52a12f215b","Type":"ContainerStarted","Data":"a8f98c323cb44b64ff3d53fd335a99fbb361d97661c9f9630fac0e48f224d46e"}
Jul 17 22:00:17.898788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.898768    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"
Jul 17 22:00:17.898898 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.898793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk" event={"ID":"669e1750-d049-4193-819d-50b724a38806","Type":"ContainerDied","Data":"20658058001b68896e4687e92cc2e5f4ff6b700594f07cbeaf7e0e0ec447df0f"}
Jul 17 22:00:17.904644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.904616    2563 scope.go:117] "RemoveContainer" containerID="f3100f7cbf4baf6bdd77f5855930f3bb5d03fe074ff1baa416bfab144d6e9449"
Jul 17 22:00:17.920792 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.920770    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"]
Jul 17 22:00:17.924573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.924551    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph8hg"]
Jul 17 22:00:17.938501 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.938438    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"]
Jul 17 22:00:17.941357 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:17.941336    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e3cf3d26-29738760-p4wfk"]
Jul 17 22:00:18.602612 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:18.602503    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669e1750-d049-4193-819d-50b724a38806" path="/var/lib/kubelet/pods/669e1750-d049-4193-819d-50b724a38806/volumes"
Jul 17 22:00:18.602936 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:18.602915    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" path="/var/lib/kubelet/pods/ee61292d-4247-4cf8-a158-3eb1abd5778d/volumes"
Jul 17 22:00:18.905979 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:18.905893    2563 generic.go:356] "Generic (PLEG): container finished" podID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerID="6365026cbb4cec45068187e554505e923196800bf6463ff35cfda3ad16dc559a" exitCode=6
Jul 17 22:00:18.906134 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:18.905976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" event={"ID":"10e92d59-d21d-4303-80cb-1f52a12f215b","Type":"ContainerDied","Data":"6365026cbb4cec45068187e554505e923196800bf6463ff35cfda3ad16dc559a"}
Jul 17 22:00:18.906188 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:18.906139    2563 scope.go:117] "RemoveContainer" containerID="6365026cbb4cec45068187e554505e923196800bf6463ff35cfda3ad16dc559a"
Jul 17 22:00:19.915365 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:19.915331    2563 generic.go:356] "Generic (PLEG): container finished" podID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerID="6e07f381b1001f90688cdc1e37e2f2ae818b7d61ce65d66a774ff13b0e7f3a49" exitCode=6
Jul 17 22:00:19.915820 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:19.915388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" event={"ID":"10e92d59-d21d-4303-80cb-1f52a12f215b","Type":"ContainerDied","Data":"6e07f381b1001f90688cdc1e37e2f2ae818b7d61ce65d66a774ff13b0e7f3a49"}
Jul 17 22:00:19.915820 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:19.915448    2563 scope.go:117] "RemoveContainer" containerID="6365026cbb4cec45068187e554505e923196800bf6463ff35cfda3ad16dc559a"
Jul 17 22:00:19.915820 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:19.915691    2563 scope.go:117] "RemoveContainer" containerID="6e07f381b1001f90688cdc1e37e2f2ae818b7d61ce65d66a774ff13b0e7f3a49"
Jul 17 22:00:19.915957 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:19.915858    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25_odh-ai-gateway-infra(10e92d59-d21d-4303-80cb-1f52a12f215b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b"
Jul 17 22:00:20.922773 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:20.922750    2563 scope.go:117] "RemoveContainer" containerID="6e07f381b1001f90688cdc1e37e2f2ae818b7d61ce65d66a774ff13b0e7f3a49"
Jul 17 22:00:20.923134 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:20.922925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25_odh-ai-gateway-infra(10e92d59-d21d-4303-80cb-1f52a12f215b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b"
Jul 17 22:00:26.599090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:26.599045    2563 scope.go:117] "RemoveContainer" containerID="0f46257f3f1f74db1731fbeb8b430ed10b270943482b42ca09cee5c8c06705cf"
Jul 17 22:00:26.953151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:26.953108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" event={"ID":"a230b4ac-dcdc-4cc2-a633-507bf2358f9a","Type":"ContainerStarted","Data":"8de01f900082e050b6aea8f16d9c7dcd42b2fc84c53d1093b3fad11bf266fe99"}
Jul 17 22:00:26.969661 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:26.969591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" podStartSLOduration=17.969577695 podStartE2EDuration="17.969577695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:26.968951178 +0000 UTC m=+1990.881191469" watchObservedRunningTime="2026-07-17 22:00:26.969577695 +0000 UTC m=+1990.881817984"
Jul 17 22:00:27.960571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:27.960540    2563 generic.go:356] "Generic (PLEG): container finished" podID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerID="8de01f900082e050b6aea8f16d9c7dcd42b2fc84c53d1093b3fad11bf266fe99" exitCode=6
Jul 17 22:00:27.961012 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:27.960640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" event={"ID":"a230b4ac-dcdc-4cc2-a633-507bf2358f9a","Type":"ContainerDied","Data":"8de01f900082e050b6aea8f16d9c7dcd42b2fc84c53d1093b3fad11bf266fe99"}
Jul 17 22:00:27.961012 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:27.960708    2563 scope.go:117] "RemoveContainer" containerID="0f46257f3f1f74db1731fbeb8b430ed10b270943482b42ca09cee5c8c06705cf"
Jul 17 22:00:27.961110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:27.961017    2563 scope.go:117] "RemoveContainer" containerID="8de01f900082e050b6aea8f16d9c7dcd42b2fc84c53d1093b3fad11bf266fe99"
Jul 17 22:00:27.961253 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:27.961226    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w_odh-ai-gateway-infra(a230b4ac-dcdc-4cc2-a633-507bf2358f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a"
Jul 17 22:00:27.981765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:27.981732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"]
Jul 17 22:00:29.112931 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.112909    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:29.182293 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.182264    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a230b4ac-dcdc-4cc2-a633-507bf2358f9a-maas-api-service-ca\") pod \"a230b4ac-dcdc-4cc2-a633-507bf2358f9a\" (UID: \"a230b4ac-dcdc-4cc2-a633-507bf2358f9a\") "
Jul 17 22:00:29.182681 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.182654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a230b4ac-dcdc-4cc2-a633-507bf2358f9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a230b4ac-dcdc-4cc2-a633-507bf2358f9a" (UID: "a230b4ac-dcdc-4cc2-a633-507bf2358f9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:29.283450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.283365    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a230b4ac-dcdc-4cc2-a633-507bf2358f9a-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:00:29.972614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.972567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w" event={"ID":"a230b4ac-dcdc-4cc2-a633-507bf2358f9a","Type":"ContainerDied","Data":"16f1a7ba55d09751dbe933622c0b56279ab34cdb1b2c397c06af4adb578b92ec"}
Jul 17 22:00:29.972800 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.972664    2563 scope.go:117] "RemoveContainer" containerID="8de01f900082e050b6aea8f16d9c7dcd42b2fc84c53d1093b3fad11bf266fe99"
Jul 17 22:00:29.972800 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.972582    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"
Jul 17 22:00:29.995066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.995034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"]
Jul 17 22:00:29.997281 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:29.997257    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9fv4w"]
Jul 17 22:00:30.602739 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:30.602698    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" path="/var/lib/kubelet/pods/a230b4ac-dcdc-4cc2-a633-507bf2358f9a/volumes"
Jul 17 22:00:34.596971 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:34.596931    2563 scope.go:117] "RemoveContainer" containerID="6e07f381b1001f90688cdc1e37e2f2ae818b7d61ce65d66a774ff13b0e7f3a49"
Jul 17 22:00:34.999545 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:34.999512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" event={"ID":"10e92d59-d21d-4303-80cb-1f52a12f215b","Type":"ContainerStarted","Data":"7f8c28a163095acd87dbc42254f3c5dd2a7322caebeea64ceb5f36d8c1edc076"}
Jul 17 22:00:36.005941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:36.005900    2563 generic.go:356] "Generic (PLEG): container finished" podID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerID="7f8c28a163095acd87dbc42254f3c5dd2a7322caebeea64ceb5f36d8c1edc076" exitCode=6
Jul 17 22:00:36.006359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:36.005972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" event={"ID":"10e92d59-d21d-4303-80cb-1f52a12f215b","Type":"ContainerDied","Data":"7f8c28a163095acd87dbc42254f3c5dd2a7322caebeea64ceb5f36d8c1edc076"}
Jul 17 22:00:36.006359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:36.006041    2563 scope.go:117] "RemoveContainer" containerID="6e07f381b1001f90688cdc1e37e2f2ae818b7d61ce65d66a774ff13b0e7f3a49"
Jul 17 22:00:36.006359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:36.006297    2563 scope.go:117] "RemoveContainer" containerID="7f8c28a163095acd87dbc42254f3c5dd2a7322caebeea64ceb5f36d8c1edc076"
Jul 17 22:00:36.006549 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:36.006525    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25_odh-ai-gateway-infra(10e92d59-d21d-4303-80cb-1f52a12f215b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b"
Jul 17 22:00:36.032883 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:36.032844    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"]
Jul 17 22:00:37.150557 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:37.150533    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:37.251699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:37.251670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e92d59-d21d-4303-80cb-1f52a12f215b-maas-api-service-ca\") pod \"10e92d59-d21d-4303-80cb-1f52a12f215b\" (UID: \"10e92d59-d21d-4303-80cb-1f52a12f215b\") "
Jul 17 22:00:37.252020 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:37.251998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e92d59-d21d-4303-80cb-1f52a12f215b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10e92d59-d21d-4303-80cb-1f52a12f215b" (UID: "10e92d59-d21d-4303-80cb-1f52a12f215b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:37.352864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:37.352784    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e92d59-d21d-4303-80cb-1f52a12f215b-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:00:38.017838 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:38.017806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25" event={"ID":"10e92d59-d21d-4303-80cb-1f52a12f215b","Type":"ContainerDied","Data":"a8f98c323cb44b64ff3d53fd335a99fbb361d97661c9f9630fac0e48f224d46e"}
Jul 17 22:00:38.017838 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:38.017829    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"
Jul 17 22:00:38.018094 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:38.017867    2563 scope.go:117] "RemoveContainer" containerID="7f8c28a163095acd87dbc42254f3c5dd2a7322caebeea64ceb5f36d8c1edc076"
Jul 17 22:00:38.039812 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:38.039787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"]
Jul 17 22:00:38.047093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:38.047071    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rbc25"]
Jul 17 22:00:38.602871 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:38.602839    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" path="/var/lib/kubelet/pods/10e92d59-d21d-4303-80cb-1f52a12f215b/volumes"
Jul 17 22:00:55.152114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152079    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"]
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152456    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152468    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="669e1750-d049-4193-819d-50b724a38806" containerName="cleanup"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152479    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152491    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152496    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152508    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152516    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152522    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152527    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152535    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152540    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:55.152614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152551    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee61292d-4247-4cf8-a158-3eb1abd5778d" containerName="revoke-keys"
Jul 17 22:00:55.153052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152640    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.153052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152650    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:00:55.153052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.153052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152669    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.153052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.152678    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:00:55.155728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.155713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:00:55.161356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.161333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:00:55.176689 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.176666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"]
Jul 17 22:00:55.212136 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.212099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb92e1c5-8176-46d9-8e42-a690fd0e86bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx\" (UID: \"eb92e1c5-8176-46d9-8e42-a690fd0e86bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:00:55.313416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.313378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb92e1c5-8176-46d9-8e42-a690fd0e86bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx\" (UID: \"eb92e1c5-8176-46d9-8e42-a690fd0e86bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:00:55.314034 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.314011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb92e1c5-8176-46d9-8e42-a690fd0e86bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx\" (UID: \"eb92e1c5-8176-46d9-8e42-a690fd0e86bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:00:55.466674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.466643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:00:55.605060 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:55.604993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"]
Jul 17 22:00:55.607587 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:00:55.607563    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb92e1c5_8176_46d9_8e42_a690fd0e86bf.slice/crio-3435cf909e290ca5a30f837380b55d4bc340799daadc847c7e4b96a6bf44fb26 WatchSource:0}: Error finding container 3435cf909e290ca5a30f837380b55d4bc340799daadc847c7e4b96a6bf44fb26: Status 404 returned error can't find the container with id 3435cf909e290ca5a30f837380b55d4bc340799daadc847c7e4b96a6bf44fb26
Jul 17 22:00:56.103715 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:56.103623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" event={"ID":"eb92e1c5-8176-46d9-8e42-a690fd0e86bf","Type":"ContainerStarted","Data":"5a680f5997e5ffd40f3213df34abeebca78c9aee3f47bdd5414663dddf9954ff"}
Jul 17 22:00:56.103715 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:56.103688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" event={"ID":"eb92e1c5-8176-46d9-8e42-a690fd0e86bf","Type":"ContainerStarted","Data":"3435cf909e290ca5a30f837380b55d4bc340799daadc847c7e4b96a6bf44fb26"}
Jul 17 22:00:56.119231 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:56.119184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" podStartSLOduration=1.1191702590000001 podStartE2EDuration="1.119170259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:56.117675039 +0000 UTC m=+2020.029915328" watchObservedRunningTime="2026-07-17 22:00:56.119170259 +0000 UTC m=+2020.031410587"
Jul 17 22:00:57.109716 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:57.109686    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerID="5a680f5997e5ffd40f3213df34abeebca78c9aee3f47bdd5414663dddf9954ff" exitCode=7
Jul 17 22:00:57.110100 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:57.109742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" event={"ID":"eb92e1c5-8176-46d9-8e42-a690fd0e86bf","Type":"ContainerDied","Data":"5a680f5997e5ffd40f3213df34abeebca78c9aee3f47bdd5414663dddf9954ff"}
Jul 17 22:00:57.110100 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:57.109973    2563 scope.go:117] "RemoveContainer" containerID="5a680f5997e5ffd40f3213df34abeebca78c9aee3f47bdd5414663dddf9954ff"
Jul 17 22:00:58.116202 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:58.116160    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerID="df69e516f505fb6a0bf0bcbb60534e44f38135bfe5218c51b56c39a95688096b" exitCode=7
Jul 17 22:00:58.116701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:58.116220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" event={"ID":"eb92e1c5-8176-46d9-8e42-a690fd0e86bf","Type":"ContainerDied","Data":"df69e516f505fb6a0bf0bcbb60534e44f38135bfe5218c51b56c39a95688096b"}
Jul 17 22:00:58.116701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:58.116270    2563 scope.go:117] "RemoveContainer" containerID="5a680f5997e5ffd40f3213df34abeebca78c9aee3f47bdd5414663dddf9954ff"
Jul 17 22:00:58.116701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:58.116504    2563 scope.go:117] "RemoveContainer" containerID="df69e516f505fb6a0bf0bcbb60534e44f38135bfe5218c51b56c39a95688096b"
Jul 17 22:00:58.116861 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:58.116757    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx_odh-ai-gateway-infra(eb92e1c5-8176-46d9-8e42-a690fd0e86bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf"
Jul 17 22:00:59.122342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:00:59.122308    2563 scope.go:117] "RemoveContainer" containerID="df69e516f505fb6a0bf0bcbb60534e44f38135bfe5218c51b56c39a95688096b"
Jul 17 22:00:59.122826 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:00:59.122552    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx_odh-ai-gateway-infra(eb92e1c5-8176-46d9-8e42-a690fd0e86bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf"
Jul 17 22:01:09.471858 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.471819    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"]
Jul 17 22:01:09.472254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.472214    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:01:09.472254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.472226    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e92d59-d21d-4303-80cb-1f52a12f215b" containerName="revoke-keys"
Jul 17 22:01:09.472329 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.472288    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a230b4ac-dcdc-4cc2-a633-507bf2358f9a" containerName="revoke-keys"
Jul 17 22:01:09.476566 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.476549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:09.482571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.482546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"]
Jul 17 22:01:09.543343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.543305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34454856-abfc-474c-9639-a1f7e2733978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2\" (UID: \"34454856-abfc-474c-9639-a1f7e2733978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:09.643863 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.643823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34454856-abfc-474c-9639-a1f7e2733978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2\" (UID: \"34454856-abfc-474c-9639-a1f7e2733978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:09.644477 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.644449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34454856-abfc-474c-9639-a1f7e2733978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2\" (UID: \"34454856-abfc-474c-9639-a1f7e2733978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:09.788165 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.788071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:09.928317 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:09.928286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"]
Jul 17 22:01:09.933377 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:01:09.933351    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34454856_abfc_474c_9639_a1f7e2733978.slice/crio-08a2e65cd127e554cdbbba7f3c67c40fad274751dcbd26d42df84af0bc7c0a4a WatchSource:0}: Error finding container 08a2e65cd127e554cdbbba7f3c67c40fad274751dcbd26d42df84af0bc7c0a4a: Status 404 returned error can't find the container with id 08a2e65cd127e554cdbbba7f3c67c40fad274751dcbd26d42df84af0bc7c0a4a
Jul 17 22:01:10.175321 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:10.175296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" event={"ID":"34454856-abfc-474c-9639-a1f7e2733978","Type":"ContainerStarted","Data":"08a2e65cd127e554cdbbba7f3c67c40fad274751dcbd26d42df84af0bc7c0a4a"}
Jul 17 22:01:11.184804 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:11.184758    2563 generic.go:356] "Generic (PLEG): container finished" podID="34454856-abfc-474c-9639-a1f7e2733978" containerID="b05bce9cdd653310a494bbb781e2e9dfcde58f5a6bbcefc58e3fdb29b8c315fb" exitCode=6
Jul 17 22:01:11.185282 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:11.184815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" event={"ID":"34454856-abfc-474c-9639-a1f7e2733978","Type":"ContainerDied","Data":"b05bce9cdd653310a494bbb781e2e9dfcde58f5a6bbcefc58e3fdb29b8c315fb"}
Jul 17 22:01:11.185282 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:11.185004    2563 scope.go:117] "RemoveContainer" containerID="b05bce9cdd653310a494bbb781e2e9dfcde58f5a6bbcefc58e3fdb29b8c315fb"
Jul 17 22:01:11.597271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:11.597240    2563 scope.go:117] "RemoveContainer" containerID="df69e516f505fb6a0bf0bcbb60534e44f38135bfe5218c51b56c39a95688096b"
Jul 17 22:01:12.193021 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.192993    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerID="4b36f6abed0a1de5196437cef3dfa30675d39c3eb4791c8ea091612b8bcd7657" exitCode=7
Jul 17 22:01:12.193423 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.193070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" event={"ID":"eb92e1c5-8176-46d9-8e42-a690fd0e86bf","Type":"ContainerDied","Data":"4b36f6abed0a1de5196437cef3dfa30675d39c3eb4791c8ea091612b8bcd7657"}
Jul 17 22:01:12.193423 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.193127    2563 scope.go:117] "RemoveContainer" containerID="df69e516f505fb6a0bf0bcbb60534e44f38135bfe5218c51b56c39a95688096b"
Jul 17 22:01:12.193537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.193481    2563 scope.go:117] "RemoveContainer" containerID="4b36f6abed0a1de5196437cef3dfa30675d39c3eb4791c8ea091612b8bcd7657"
Jul 17 22:01:12.193757 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:12.193737    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx_odh-ai-gateway-infra(eb92e1c5-8176-46d9-8e42-a690fd0e86bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf"
Jul 17 22:01:12.195180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.195162    2563 generic.go:356] "Generic (PLEG): container finished" podID="34454856-abfc-474c-9639-a1f7e2733978" containerID="36b782b0c9144006bf81ef481ee61d4ac1f45bbec1ad79c518048ca9d0679f6c" exitCode=6
Jul 17 22:01:12.195261 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.195188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" event={"ID":"34454856-abfc-474c-9639-a1f7e2733978","Type":"ContainerDied","Data":"36b782b0c9144006bf81ef481ee61d4ac1f45bbec1ad79c518048ca9d0679f6c"}
Jul 17 22:01:12.195380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.195368    2563 scope.go:117] "RemoveContainer" containerID="36b782b0c9144006bf81ef481ee61d4ac1f45bbec1ad79c518048ca9d0679f6c"
Jul 17 22:01:12.195529 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:12.195514    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2_odh-ai-gateway-infra(34454856-abfc-474c-9639-a1f7e2733978)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" podUID="34454856-abfc-474c-9639-a1f7e2733978"
Jul 17 22:01:12.218904 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:12.218882    2563 scope.go:117] "RemoveContainer" containerID="b05bce9cdd653310a494bbb781e2e9dfcde58f5a6bbcefc58e3fdb29b8c315fb"
Jul 17 22:01:13.202687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:13.202656    2563 scope.go:117] "RemoveContainer" containerID="36b782b0c9144006bf81ef481ee61d4ac1f45bbec1ad79c518048ca9d0679f6c"
Jul 17 22:01:13.203124 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:13.202875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2_odh-ai-gateway-infra(34454856-abfc-474c-9639-a1f7e2733978)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" podUID="34454856-abfc-474c-9639-a1f7e2733978"
Jul 17 22:01:13.218261 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:13.218232    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"]
Jul 17 22:01:13.354485 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:13.354461    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:01:13.478097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:13.478011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb92e1c5-8176-46d9-8e42-a690fd0e86bf-maas-api-service-ca\") pod \"eb92e1c5-8176-46d9-8e42-a690fd0e86bf\" (UID: \"eb92e1c5-8176-46d9-8e42-a690fd0e86bf\") "
Jul 17 22:01:13.478407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:13.478382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb92e1c5-8176-46d9-8e42-a690fd0e86bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb92e1c5-8176-46d9-8e42-a690fd0e86bf" (UID: "eb92e1c5-8176-46d9-8e42-a690fd0e86bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:13.579349 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:13.579297    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb92e1c5-8176-46d9-8e42-a690fd0e86bf-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:01:14.210271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:14.210239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx" event={"ID":"eb92e1c5-8176-46d9-8e42-a690fd0e86bf","Type":"ContainerDied","Data":"3435cf909e290ca5a30f837380b55d4bc340799daadc847c7e4b96a6bf44fb26"}
Jul 17 22:01:14.210271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:14.210261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"
Jul 17 22:01:14.210784 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:14.210298    2563 scope.go:117] "RemoveContainer" containerID="4b36f6abed0a1de5196437cef3dfa30675d39c3eb4791c8ea091612b8bcd7657"
Jul 17 22:01:14.235813 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:14.235790    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"]
Jul 17 22:01:14.241419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:14.241394    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-2jfhx"]
Jul 17 22:01:14.602716 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:14.602640    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" path="/var/lib/kubelet/pods/eb92e1c5-8176-46d9-8e42-a690fd0e86bf/volumes"
Jul 17 22:01:17.265475 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.265437    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"]
Jul 17 22:01:17.266059 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266037    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266061    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266105    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266114    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266222    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.266384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.266231    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb92e1c5-8176-46d9-8e42-a690fd0e86bf" containerName="revoke-keys"
Jul 17 22:01:17.270901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.270879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:17.275917 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.275889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"]
Jul 17 22:01:17.410824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.410783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl\" (UID: \"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:17.511841 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.511800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl\" (UID: \"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:17.512490 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.512469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl\" (UID: \"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:17.583398 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.583306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:17.723452 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:17.723430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"]
Jul 17 22:01:17.725650 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:01:17.725617    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9f4c91_84b7_4fc0_b7f1_2a2c2faf9d07.slice/crio-ea8045247316c7b76d372890dc9b380f525cafdcadeedced90bfa45ca3218e92 WatchSource:0}: Error finding container ea8045247316c7b76d372890dc9b380f525cafdcadeedced90bfa45ca3218e92: Status 404 returned error can't find the container with id ea8045247316c7b76d372890dc9b380f525cafdcadeedced90bfa45ca3218e92
Jul 17 22:01:18.230464 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:18.230369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" event={"ID":"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07","Type":"ContainerStarted","Data":"e4b92fbeaa0b1c0ab3d2ddf8c0687b20ba1d8bf9d2fa108150a16ee16ba14d5e"}
Jul 17 22:01:18.230464 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:18.230420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" event={"ID":"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07","Type":"ContainerStarted","Data":"ea8045247316c7b76d372890dc9b380f525cafdcadeedced90bfa45ca3218e92"}
Jul 17 22:01:18.246789 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:18.246747    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" podStartSLOduration=1.246732105 podStartE2EDuration="1.246732105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:18.246163194 +0000 UTC m=+2042.158403484" watchObservedRunningTime="2026-07-17 22:01:18.246732105 +0000 UTC m=+2042.158972393"
Jul 17 22:01:19.236589 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:19.236552    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerID="e4b92fbeaa0b1c0ab3d2ddf8c0687b20ba1d8bf9d2fa108150a16ee16ba14d5e" exitCode=6
Jul 17 22:01:19.237005 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:19.236640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" event={"ID":"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07","Type":"ContainerDied","Data":"e4b92fbeaa0b1c0ab3d2ddf8c0687b20ba1d8bf9d2fa108150a16ee16ba14d5e"}
Jul 17 22:01:19.237005 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:19.236827    2563 scope.go:117] "RemoveContainer" containerID="e4b92fbeaa0b1c0ab3d2ddf8c0687b20ba1d8bf9d2fa108150a16ee16ba14d5e"
Jul 17 22:01:20.243509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:20.243473    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerID="2928f3a62695c012c0fe795e97d0f03ba9f4ec3d23ccc3d3e580183fd9932234" exitCode=6
Jul 17 22:01:20.244029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:20.243535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" event={"ID":"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07","Type":"ContainerDied","Data":"2928f3a62695c012c0fe795e97d0f03ba9f4ec3d23ccc3d3e580183fd9932234"}
Jul 17 22:01:20.244029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:20.243584    2563 scope.go:117] "RemoveContainer" containerID="e4b92fbeaa0b1c0ab3d2ddf8c0687b20ba1d8bf9d2fa108150a16ee16ba14d5e"
Jul 17 22:01:20.244029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:20.243809    2563 scope.go:117] "RemoveContainer" containerID="2928f3a62695c012c0fe795e97d0f03ba9f4ec3d23ccc3d3e580183fd9932234"
Jul 17 22:01:20.244214 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:20.244074    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl_odh-ai-gateway-infra(8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07"
Jul 17 22:01:21.250226 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:21.250203    2563 scope.go:117] "RemoveContainer" containerID="2928f3a62695c012c0fe795e97d0f03ba9f4ec3d23ccc3d3e580183fd9932234"
Jul 17 22:01:21.250587 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:21.250387    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl_odh-ai-gateway-infra(8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07"
Jul 17 22:01:27.597160 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:27.597121    2563 scope.go:117] "RemoveContainer" containerID="36b782b0c9144006bf81ef481ee61d4ac1f45bbec1ad79c518048ca9d0679f6c"
Jul 17 22:01:28.282441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:28.282406    2563 generic.go:356] "Generic (PLEG): container finished" podID="34454856-abfc-474c-9639-a1f7e2733978" containerID="6fb7f15f85215676b9b2d4d80cf656995a4c553041fc222f81f3b1f4cccd0b1e" exitCode=6
Jul 17 22:01:28.282618 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:28.282475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" event={"ID":"34454856-abfc-474c-9639-a1f7e2733978","Type":"ContainerDied","Data":"6fb7f15f85215676b9b2d4d80cf656995a4c553041fc222f81f3b1f4cccd0b1e"}
Jul 17 22:01:28.282618 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:28.282568    2563 scope.go:117] "RemoveContainer" containerID="36b782b0c9144006bf81ef481ee61d4ac1f45bbec1ad79c518048ca9d0679f6c"
Jul 17 22:01:28.282943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:28.282913    2563 scope.go:117] "RemoveContainer" containerID="6fb7f15f85215676b9b2d4d80cf656995a4c553041fc222f81f3b1f4cccd0b1e"
Jul 17 22:01:28.283157 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:28.283137    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2_odh-ai-gateway-infra(34454856-abfc-474c-9639-a1f7e2733978)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" podUID="34454856-abfc-474c-9639-a1f7e2733978"
Jul 17 22:01:29.309053 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:29.309017    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"]
Jul 17 22:01:29.452836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:29.452812    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:29.620645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:29.620532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34454856-abfc-474c-9639-a1f7e2733978-maas-api-service-ca\") pod \"34454856-abfc-474c-9639-a1f7e2733978\" (UID: \"34454856-abfc-474c-9639-a1f7e2733978\") "
Jul 17 22:01:29.620916 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:29.620887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34454856-abfc-474c-9639-a1f7e2733978-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34454856-abfc-474c-9639-a1f7e2733978" (UID: "34454856-abfc-474c-9639-a1f7e2733978"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:29.721757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:29.721702    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34454856-abfc-474c-9639-a1f7e2733978-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:01:30.296571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:30.296536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2" event={"ID":"34454856-abfc-474c-9639-a1f7e2733978","Type":"ContainerDied","Data":"08a2e65cd127e554cdbbba7f3c67c40fad274751dcbd26d42df84af0bc7c0a4a"}
Jul 17 22:01:30.296571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:30.296564    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"
Jul 17 22:01:30.296841 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:30.296617    2563 scope.go:117] "RemoveContainer" containerID="6fb7f15f85215676b9b2d4d80cf656995a4c553041fc222f81f3b1f4cccd0b1e"
Jul 17 22:01:30.316765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:30.316739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"]
Jul 17 22:01:30.320449 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:30.320425    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pb7j2"]
Jul 17 22:01:30.603273 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:30.603192    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34454856-abfc-474c-9639-a1f7e2733978" path="/var/lib/kubelet/pods/34454856-abfc-474c-9639-a1f7e2733978/volumes"
Jul 17 22:01:35.596680 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:35.596651    2563 scope.go:117] "RemoveContainer" containerID="2928f3a62695c012c0fe795e97d0f03ba9f4ec3d23ccc3d3e580183fd9932234"
Jul 17 22:01:36.334954 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:36.334920    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerID="100bd8169f6ff330f239032995ae1a376c3d7030d1c096f8614b7b1dfd007653" exitCode=6
Jul 17 22:01:36.335120 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:36.334994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" event={"ID":"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07","Type":"ContainerDied","Data":"100bd8169f6ff330f239032995ae1a376c3d7030d1c096f8614b7b1dfd007653"}
Jul 17 22:01:36.335120 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:36.335055    2563 scope.go:117] "RemoveContainer" containerID="2928f3a62695c012c0fe795e97d0f03ba9f4ec3d23ccc3d3e580183fd9932234"
Jul 17 22:01:36.335407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:36.335384    2563 scope.go:117] "RemoveContainer" containerID="100bd8169f6ff330f239032995ae1a376c3d7030d1c096f8614b7b1dfd007653"
Jul 17 22:01:36.335657 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:36.335639    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl_odh-ai-gateway-infra(8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07"
Jul 17 22:01:37.361727 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:37.361695    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"]
Jul 17 22:01:37.503364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:37.503338    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:37.691171 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:37.691140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07-maas-api-service-ca\") pod \"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07\" (UID: \"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07\") "
Jul 17 22:01:37.691513 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:37.691488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" (UID: "8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:37.792066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:37.791993    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:01:38.346079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:38.346045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl" event={"ID":"8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07","Type":"ContainerDied","Data":"ea8045247316c7b76d372890dc9b380f525cafdcadeedced90bfa45ca3218e92"}
Jul 17 22:01:38.346079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:38.346064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"
Jul 17 22:01:38.346329 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:38.346101    2563 scope.go:117] "RemoveContainer" containerID="100bd8169f6ff330f239032995ae1a376c3d7030d1c096f8614b7b1dfd007653"
Jul 17 22:01:38.368634 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:38.368591    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"]
Jul 17 22:01:38.373688 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:38.373667    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-b9bbl"]
Jul 17 22:01:38.602870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:38.602797    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" path="/var/lib/kubelet/pods/8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07/volumes"
Jul 17 22:01:45.167727 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.167697    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"]
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168036    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168060    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168065    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168072    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168077    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168086    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:01:45.168126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168091    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:01:45.168399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168154    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:01:45.168399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168163    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168170    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168175    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:01:45.168399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.168190    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:01:45.172706 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.172681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:01:45.175289 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.175226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:01:45.177876 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.177854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"]
Jul 17 22:01:45.243019 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.242985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29\" (UID: \"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:01:45.343976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.343940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29\" (UID: \"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:01:45.344535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.344516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29\" (UID: \"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:01:45.485072 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.484980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:01:45.623051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:45.622965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"]
Jul 17 22:01:45.625733 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:01:45.625705    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfb5ae2_2bdd_4ffe_a8e1_ef173bc46f0d.slice/crio-fba5bab48c94c8d1891295626e761ff04ac485518aeabd9b1942ef561639bf5b WatchSource:0}: Error finding container fba5bab48c94c8d1891295626e761ff04ac485518aeabd9b1942ef561639bf5b: Status 404 returned error can't find the container with id fba5bab48c94c8d1891295626e761ff04ac485518aeabd9b1942ef561639bf5b
Jul 17 22:01:46.391622 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:46.391545    2563 generic.go:356] "Generic (PLEG): container finished" podID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerID="f3467bba124105db06f33d87cfe298e755385ef2f0939a3959f7c5f2ea59b102" exitCode=7
Jul 17 22:01:46.392106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:46.391624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" event={"ID":"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d","Type":"ContainerDied","Data":"f3467bba124105db06f33d87cfe298e755385ef2f0939a3959f7c5f2ea59b102"}
Jul 17 22:01:46.392106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:46.391721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" event={"ID":"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d","Type":"ContainerStarted","Data":"fba5bab48c94c8d1891295626e761ff04ac485518aeabd9b1942ef561639bf5b"}
Jul 17 22:01:46.392106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:46.391935    2563 scope.go:117] "RemoveContainer" containerID="f3467bba124105db06f33d87cfe298e755385ef2f0939a3959f7c5f2ea59b102"
Jul 17 22:01:47.398525 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:47.398432    2563 generic.go:356] "Generic (PLEG): container finished" podID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerID="af4524481a3076e6e5e0c0fa8369424122985ebb7cb2cd14d8be5ffa267eed5f" exitCode=7
Jul 17 22:01:47.398525 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:47.398475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" event={"ID":"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d","Type":"ContainerDied","Data":"af4524481a3076e6e5e0c0fa8369424122985ebb7cb2cd14d8be5ffa267eed5f"}
Jul 17 22:01:47.398525 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:47.398523    2563 scope.go:117] "RemoveContainer" containerID="f3467bba124105db06f33d87cfe298e755385ef2f0939a3959f7c5f2ea59b102"
Jul 17 22:01:47.399062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:47.398755    2563 scope.go:117] "RemoveContainer" containerID="af4524481a3076e6e5e0c0fa8369424122985ebb7cb2cd14d8be5ffa267eed5f"
Jul 17 22:01:47.399062 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:47.398983    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29_odh-ai-gateway-infra(7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d"
Jul 17 22:01:48.404424 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:48.404397    2563 scope.go:117] "RemoveContainer" containerID="af4524481a3076e6e5e0c0fa8369424122985ebb7cb2cd14d8be5ffa267eed5f"
Jul 17 22:01:48.404852 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:01:48.404580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29_odh-ai-gateway-infra(7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d"
Jul 17 22:01:59.596542 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:01:59.596510    2563 scope.go:117] "RemoveContainer" containerID="af4524481a3076e6e5e0c0fa8369424122985ebb7cb2cd14d8be5ffa267eed5f"
Jul 17 22:02:00.464471 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:00.464438    2563 generic.go:356] "Generic (PLEG): container finished" podID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerID="c4efbe4131695a5b4a452ca7da48a6dd0185ad68c4ef83a0a87ca672eb177fdf" exitCode=7
Jul 17 22:02:00.464664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:00.464487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" event={"ID":"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d","Type":"ContainerDied","Data":"c4efbe4131695a5b4a452ca7da48a6dd0185ad68c4ef83a0a87ca672eb177fdf"}
Jul 17 22:02:00.464664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:00.464535    2563 scope.go:117] "RemoveContainer" containerID="af4524481a3076e6e5e0c0fa8369424122985ebb7cb2cd14d8be5ffa267eed5f"
Jul 17 22:02:00.464872 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:00.464856    2563 scope.go:117] "RemoveContainer" containerID="c4efbe4131695a5b4a452ca7da48a6dd0185ad68c4ef83a0a87ca672eb177fdf"
Jul 17 22:02:00.465082 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:00.465062    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29_odh-ai-gateway-infra(7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d"
Jul 17 22:02:01.488701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:01.488668    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"]
Jul 17 22:02:01.626995 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:01.626965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:02:01.680979 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:01.680950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d-maas-api-service-ca\") pod \"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d\" (UID: \"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d\") "
Jul 17 22:02:01.681328 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:01.681305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" (UID: "7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:01.781877 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:01.781792    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:02:02.476971 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:02.476946    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"
Jul 17 22:02:02.477144 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:02.476945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29" event={"ID":"7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d","Type":"ContainerDied","Data":"fba5bab48c94c8d1891295626e761ff04ac485518aeabd9b1942ef561639bf5b"}
Jul 17 22:02:02.477144 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:02.477082    2563 scope.go:117] "RemoveContainer" containerID="c4efbe4131695a5b4a452ca7da48a6dd0185ad68c4ef83a0a87ca672eb177fdf"
Jul 17 22:02:02.498940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:02.498910    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"]
Jul 17 22:02:02.503051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:02.503028    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-vzr29"]
Jul 17 22:02:02.603061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:02.603033    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" path="/var/lib/kubelet/pods/7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d/volumes"
Jul 17 22:02:09.516233 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516195    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"]
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516554    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516580    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9f4c91-84b7-4fc0-b7f1-2a2c2faf9d07" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516588    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516608    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516622    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:02:09.516644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516629    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:02:09.516920 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:09.516920 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516708    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34454856-abfc-474c-9639-a1f7e2733978" containerName="revoke-keys"
Jul 17 22:02:09.516920 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.516714    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:09.521772 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.521755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:09.524232 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.524200    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:02:09.529721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.529694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"]
Jul 17 22:02:09.650488 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.650459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl\" (UID: \"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:09.751300 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.751258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl\" (UID: \"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:09.751898 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.751880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl\" (UID: \"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:09.834505 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.834422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:09.968058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:09.968035    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"]
Jul 17 22:02:09.970511 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:02:09.970480    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02934ee0_fb23_4c63_a8f8_b8e3adc0aa5f.slice/crio-305f0c4b16d48d26057c1b487c62ab5ed376b3dce263e3f6c3b23af723f6bcea WatchSource:0}: Error finding container 305f0c4b16d48d26057c1b487c62ab5ed376b3dce263e3f6c3b23af723f6bcea: Status 404 returned error can't find the container with id 305f0c4b16d48d26057c1b487c62ab5ed376b3dce263e3f6c3b23af723f6bcea
Jul 17 22:02:10.516425 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:10.516328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" event={"ID":"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f","Type":"ContainerStarted","Data":"e923b0491693780ed63c1a694ac0b0ab03298f6c7b56090cfd47881925dd6306"}
Jul 17 22:02:10.516425 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:10.516386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" event={"ID":"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f","Type":"ContainerStarted","Data":"305f0c4b16d48d26057c1b487c62ab5ed376b3dce263e3f6c3b23af723f6bcea"}
Jul 17 22:02:10.531959 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:10.531913    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" podStartSLOduration=1.5318985189999998 podStartE2EDuration="1.531898519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:10.530725363 +0000 UTC m=+2094.442965651" watchObservedRunningTime="2026-07-17 22:02:10.531898519 +0000 UTC m=+2094.444138808"
Jul 17 22:02:11.522201 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:11.522173    2563 generic.go:356] "Generic (PLEG): container finished" podID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerID="e923b0491693780ed63c1a694ac0b0ab03298f6c7b56090cfd47881925dd6306" exitCode=6
Jul 17 22:02:11.522621 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:11.522209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" event={"ID":"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f","Type":"ContainerDied","Data":"e923b0491693780ed63c1a694ac0b0ab03298f6c7b56090cfd47881925dd6306"}
Jul 17 22:02:11.522621 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:11.522456    2563 scope.go:117] "RemoveContainer" containerID="e923b0491693780ed63c1a694ac0b0ab03298f6c7b56090cfd47881925dd6306"
Jul 17 22:02:12.528364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:12.528332    2563 generic.go:356] "Generic (PLEG): container finished" podID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerID="1871e335063319c0e019f993959085670e63df873f0e9f22ec2da54f13283934" exitCode=6
Jul 17 22:02:12.528879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:12.528419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" event={"ID":"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f","Type":"ContainerDied","Data":"1871e335063319c0e019f993959085670e63df873f0e9f22ec2da54f13283934"}
Jul 17 22:02:12.528879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:12.528476    2563 scope.go:117] "RemoveContainer" containerID="e923b0491693780ed63c1a694ac0b0ab03298f6c7b56090cfd47881925dd6306"
Jul 17 22:02:12.528879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:12.528689    2563 scope.go:117] "RemoveContainer" containerID="1871e335063319c0e019f993959085670e63df873f0e9f22ec2da54f13283934"
Jul 17 22:02:12.529038 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:12.528922    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl_odh-ai-gateway-infra(02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f"
Jul 17 22:02:13.534930 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:13.534898    2563 scope.go:117] "RemoveContainer" containerID="1871e335063319c0e019f993959085670e63df873f0e9f22ec2da54f13283934"
Jul 17 22:02:13.535396 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:13.535167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl_odh-ai-gateway-infra(02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f"
Jul 17 22:02:16.694513 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:16.694486    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:02:16.697926 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:16.697906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:02:17.313081 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.313043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"]
Jul 17 22:02:17.313450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.313436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:17.313496 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.313453    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:17.313553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.313542    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfb5ae2-2bdd-4ffe-a8e1-ef173bc46f0d" containerName="revoke-keys"
Jul 17 22:02:17.318023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.318007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:17.323325 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.323295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"]
Jul 17 22:02:17.417194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.417152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84417881-2807-41d7-b21b-00e77fd2e481-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps\" (UID: \"84417881-2807-41d7-b21b-00e77fd2e481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:17.518506 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.518471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84417881-2807-41d7-b21b-00e77fd2e481-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps\" (UID: \"84417881-2807-41d7-b21b-00e77fd2e481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:17.519167 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.519145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84417881-2807-41d7-b21b-00e77fd2e481-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps\" (UID: \"84417881-2807-41d7-b21b-00e77fd2e481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:17.630812 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.630723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:17.767563 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:17.767525    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"]
Jul 17 22:02:17.770567 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:02:17.770540    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84417881_2807_41d7_b21b_00e77fd2e481.slice/crio-dcc92d3c41fbc304bb299339e124f8214c7b6fafa28d1f5f9540ef8a1b702bf9 WatchSource:0}: Error finding container dcc92d3c41fbc304bb299339e124f8214c7b6fafa28d1f5f9540ef8a1b702bf9: Status 404 returned error can't find the container with id dcc92d3c41fbc304bb299339e124f8214c7b6fafa28d1f5f9540ef8a1b702bf9
Jul 17 22:02:18.560652 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:18.560615    2563 generic.go:356] "Generic (PLEG): container finished" podID="84417881-2807-41d7-b21b-00e77fd2e481" containerID="c5f940f2bee3c2e20999258eb72a2399d6316ac1ade5018d0f1ad52fa8ae4a7d" exitCode=6
Jul 17 22:02:18.560854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:18.560691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" event={"ID":"84417881-2807-41d7-b21b-00e77fd2e481","Type":"ContainerDied","Data":"c5f940f2bee3c2e20999258eb72a2399d6316ac1ade5018d0f1ad52fa8ae4a7d"}
Jul 17 22:02:18.560854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:18.560745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" event={"ID":"84417881-2807-41d7-b21b-00e77fd2e481","Type":"ContainerStarted","Data":"dcc92d3c41fbc304bb299339e124f8214c7b6fafa28d1f5f9540ef8a1b702bf9"}
Jul 17 22:02:18.560969 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:18.560882    2563 scope.go:117] "RemoveContainer" containerID="c5f940f2bee3c2e20999258eb72a2399d6316ac1ade5018d0f1ad52fa8ae4a7d"
Jul 17 22:02:19.568928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:19.568894    2563 generic.go:356] "Generic (PLEG): container finished" podID="84417881-2807-41d7-b21b-00e77fd2e481" containerID="50e7921cfe4059df7ec8e2a7c5a53d93b10de899eb4e3a3efcc6f8eb956d0b4a" exitCode=6
Jul 17 22:02:19.569350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:19.568960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" event={"ID":"84417881-2807-41d7-b21b-00e77fd2e481","Type":"ContainerDied","Data":"50e7921cfe4059df7ec8e2a7c5a53d93b10de899eb4e3a3efcc6f8eb956d0b4a"}
Jul 17 22:02:19.569350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:19.569020    2563 scope.go:117] "RemoveContainer" containerID="c5f940f2bee3c2e20999258eb72a2399d6316ac1ade5018d0f1ad52fa8ae4a7d"
Jul 17 22:02:19.569350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:19.569303    2563 scope.go:117] "RemoveContainer" containerID="50e7921cfe4059df7ec8e2a7c5a53d93b10de899eb4e3a3efcc6f8eb956d0b4a"
Jul 17 22:02:19.569533 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:19.569510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps_odh-ai-gateway-infra(84417881-2807-41d7-b21b-00e77fd2e481)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" podUID="84417881-2807-41d7-b21b-00e77fd2e481"
Jul 17 22:02:20.575547 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:20.575518    2563 scope.go:117] "RemoveContainer" containerID="50e7921cfe4059df7ec8e2a7c5a53d93b10de899eb4e3a3efcc6f8eb956d0b4a"
Jul 17 22:02:20.575957 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:20.575738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps_odh-ai-gateway-infra(84417881-2807-41d7-b21b-00e77fd2e481)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" podUID="84417881-2807-41d7-b21b-00e77fd2e481"
Jul 17 22:02:26.599676 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:26.599623    2563 scope.go:117] "RemoveContainer" containerID="1871e335063319c0e019f993959085670e63df873f0e9f22ec2da54f13283934"
Jul 17 22:02:27.610109 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:27.610020    2563 generic.go:356] "Generic (PLEG): container finished" podID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerID="b14734034db9f48201e1ac887e90b31ff4d1d43b1a5eb9935715d21c3a4e14a3" exitCode=6
Jul 17 22:02:27.610109 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:27.610097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" event={"ID":"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f","Type":"ContainerDied","Data":"b14734034db9f48201e1ac887e90b31ff4d1d43b1a5eb9935715d21c3a4e14a3"}
Jul 17 22:02:27.610543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:27.610156    2563 scope.go:117] "RemoveContainer" containerID="1871e335063319c0e019f993959085670e63df873f0e9f22ec2da54f13283934"
Jul 17 22:02:27.610543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:27.610401    2563 scope.go:117] "RemoveContainer" containerID="b14734034db9f48201e1ac887e90b31ff4d1d43b1a5eb9935715d21c3a4e14a3"
Jul 17 22:02:27.610640 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:27.610623    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl_odh-ai-gateway-infra(02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f"
Jul 17 22:02:28.633990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:28.633950    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"]
Jul 17 22:02:28.770797 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:28.770772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:28.919560 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:28.919527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f-maas-api-service-ca\") pod \"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f\" (UID: \"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f\") "
Jul 17 22:02:28.919909 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:28.919886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" (UID: "02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:29.020858 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:29.020824    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:02:29.622077 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:29.622042    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"
Jul 17 22:02:29.622290 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:29.622046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl" event={"ID":"02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f","Type":"ContainerDied","Data":"305f0c4b16d48d26057c1b487c62ab5ed376b3dce263e3f6c3b23af723f6bcea"}
Jul 17 22:02:29.622290 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:29.622180    2563 scope.go:117] "RemoveContainer" containerID="b14734034db9f48201e1ac887e90b31ff4d1d43b1a5eb9935715d21c3a4e14a3"
Jul 17 22:02:29.644369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:29.644340    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"]
Jul 17 22:02:29.649962 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:29.649940    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vv6xl"]
Jul 17 22:02:30.602464 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:30.602431    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" path="/var/lib/kubelet/pods/02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f/volumes"
Jul 17 22:02:35.203458 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203429    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"]
Jul 17 22:02:35.203850 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203815    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.203850 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203825    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.203929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203853    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.203929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203858    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.203929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203913    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.203929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203921    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.203929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.203927    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:02:35.207180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.207163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:35.213756 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.213732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"]
Jul 17 22:02:35.375789 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.375753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2\" (UID: \"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:35.476555 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.476463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2\" (UID: \"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:35.477099 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.477080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2\" (UID: \"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:35.520437 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.520415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:35.597413 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.597385    2563 scope.go:117] "RemoveContainer" containerID="50e7921cfe4059df7ec8e2a7c5a53d93b10de899eb4e3a3efcc6f8eb956d0b4a"
Jul 17 22:02:35.662190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:35.662129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"]
Jul 17 22:02:35.664777 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:02:35.664749    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcc07c6_d2a4_425d_bfe6_31f0c3ca0056.slice/crio-fa205081a43389f77693e12512af084ad5ac6004abd693c23c44ff153a28c0ba WatchSource:0}: Error finding container fa205081a43389f77693e12512af084ad5ac6004abd693c23c44ff153a28c0ba: Status 404 returned error can't find the container with id fa205081a43389f77693e12512af084ad5ac6004abd693c23c44ff153a28c0ba
Jul 17 22:02:36.660877 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.660844    2563 generic.go:356] "Generic (PLEG): container finished" podID="84417881-2807-41d7-b21b-00e77fd2e481" containerID="d478fa4d9bd38dcd444058b0d6b10fb70c6d864451b32a8108009676e304059c" exitCode=6
Jul 17 22:02:36.661323 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.660907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" event={"ID":"84417881-2807-41d7-b21b-00e77fd2e481","Type":"ContainerDied","Data":"d478fa4d9bd38dcd444058b0d6b10fb70c6d864451b32a8108009676e304059c"}
Jul 17 22:02:36.661323 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.660963    2563 scope.go:117] "RemoveContainer" containerID="50e7921cfe4059df7ec8e2a7c5a53d93b10de899eb4e3a3efcc6f8eb956d0b4a"
Jul 17 22:02:36.661323 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.661286    2563 scope.go:117] "RemoveContainer" containerID="d478fa4d9bd38dcd444058b0d6b10fb70c6d864451b32a8108009676e304059c"
Jul 17 22:02:36.661515 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:36.661495    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps_odh-ai-gateway-infra(84417881-2807-41d7-b21b-00e77fd2e481)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" podUID="84417881-2807-41d7-b21b-00e77fd2e481"
Jul 17 22:02:36.662675 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.662649    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerID="03398edcc70ec57f1caa80796bc23dc605c9c2236fa06be0cc90611f4c6aebb8" exitCode=7
Jul 17 22:02:36.662771 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.662692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" event={"ID":"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056","Type":"ContainerDied","Data":"03398edcc70ec57f1caa80796bc23dc605c9c2236fa06be0cc90611f4c6aebb8"}
Jul 17 22:02:36.662771 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.662740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" event={"ID":"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056","Type":"ContainerStarted","Data":"fa205081a43389f77693e12512af084ad5ac6004abd693c23c44ff153a28c0ba"}
Jul 17 22:02:36.662922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:36.662909    2563 scope.go:117] "RemoveContainer" containerID="03398edcc70ec57f1caa80796bc23dc605c9c2236fa06be0cc90611f4c6aebb8"
Jul 17 22:02:37.670733 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.670698    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerID="b6221befd0de45d6ea59d4aac9c02638bcb2984566fe4dabdf1bd3e39e8fed92" exitCode=7
Jul 17 22:02:37.671210 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.670750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" event={"ID":"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056","Type":"ContainerDied","Data":"b6221befd0de45d6ea59d4aac9c02638bcb2984566fe4dabdf1bd3e39e8fed92"}
Jul 17 22:02:37.671210 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.670799    2563 scope.go:117] "RemoveContainer" containerID="03398edcc70ec57f1caa80796bc23dc605c9c2236fa06be0cc90611f4c6aebb8"
Jul 17 22:02:37.671210 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.671037    2563 scope.go:117] "RemoveContainer" containerID="b6221befd0de45d6ea59d4aac9c02638bcb2984566fe4dabdf1bd3e39e8fed92"
Jul 17 22:02:37.671374 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:37.671268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2_odh-ai-gateway-infra(5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056"
Jul 17 22:02:37.683402 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.683380    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"]
Jul 17 22:02:37.836158 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.836133    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:37.898683 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.898653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84417881-2807-41d7-b21b-00e77fd2e481-maas-api-service-ca\") pod \"84417881-2807-41d7-b21b-00e77fd2e481\" (UID: \"84417881-2807-41d7-b21b-00e77fd2e481\") "
Jul 17 22:02:37.898999 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.898978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84417881-2807-41d7-b21b-00e77fd2e481-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84417881-2807-41d7-b21b-00e77fd2e481" (UID: "84417881-2807-41d7-b21b-00e77fd2e481"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:38.000015 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:37.999929    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84417881-2807-41d7-b21b-00e77fd2e481-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:02:38.676933 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:38.676908    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"
Jul 17 22:02:38.676933 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:38.676921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps" event={"ID":"84417881-2807-41d7-b21b-00e77fd2e481","Type":"ContainerDied","Data":"dcc92d3c41fbc304bb299339e124f8214c7b6fafa28d1f5f9540ef8a1b702bf9"}
Jul 17 22:02:38.677424 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:38.676981    2563 scope.go:117] "RemoveContainer" containerID="d478fa4d9bd38dcd444058b0d6b10fb70c6d864451b32a8108009676e304059c"
Jul 17 22:02:38.679190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:38.679152    2563 scope.go:117] "RemoveContainer" containerID="b6221befd0de45d6ea59d4aac9c02638bcb2984566fe4dabdf1bd3e39e8fed92"
Jul 17 22:02:38.679422 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:38.679391    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2_odh-ai-gateway-infra(5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056"
Jul 17 22:02:38.706758 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:38.706734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"]
Jul 17 22:02:38.712496 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:38.712465    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-r4mps"]
Jul 17 22:02:40.602451 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:40.602420    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84417881-2807-41d7-b21b-00e77fd2e481" path="/var/lib/kubelet/pods/84417881-2807-41d7-b21b-00e77fd2e481/volumes"
Jul 17 22:02:50.597401 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:50.597369    2563 scope.go:117] "RemoveContainer" containerID="b6221befd0de45d6ea59d4aac9c02638bcb2984566fe4dabdf1bd3e39e8fed92"
Jul 17 22:02:51.746570 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:51.746539    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerID="3724bd781c802c103559e6f8781017b995c00c9d20e0e059f59571621d44a3a2" exitCode=7
Jul 17 22:02:51.746991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:51.746621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" event={"ID":"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056","Type":"ContainerDied","Data":"3724bd781c802c103559e6f8781017b995c00c9d20e0e059f59571621d44a3a2"}
Jul 17 22:02:51.746991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:51.746679    2563 scope.go:117] "RemoveContainer" containerID="b6221befd0de45d6ea59d4aac9c02638bcb2984566fe4dabdf1bd3e39e8fed92"
Jul 17 22:02:51.747079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:51.746986    2563 scope.go:117] "RemoveContainer" containerID="3724bd781c802c103559e6f8781017b995c00c9d20e0e059f59571621d44a3a2"
Jul 17 22:02:51.747231 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:02:51.747206    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2_odh-ai-gateway-infra(5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056"
Jul 17 22:02:52.771703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:52.771668    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"]
Jul 17 22:02:52.912485 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:52.912459    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:53.030509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.030433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056-maas-api-service-ca\") pod \"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056\" (UID: \"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056\") "
Jul 17 22:02:53.030813 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.030789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" (UID: "5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:53.131940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.131903    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:02:53.758670 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.758637    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"
Jul 17 22:02:53.758670 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.758649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2" event={"ID":"5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056","Type":"ContainerDied","Data":"fa205081a43389f77693e12512af084ad5ac6004abd693c23c44ff153a28c0ba"}
Jul 17 22:02:53.758892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.758716    2563 scope.go:117] "RemoveContainer" containerID="3724bd781c802c103559e6f8781017b995c00c9d20e0e059f59571621d44a3a2"
Jul 17 22:02:53.778668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.778643    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"]
Jul 17 22:02:53.784280 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:53.784257    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9cvl2"]
Jul 17 22:02:54.602876 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:02:54.602844    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" path="/var/lib/kubelet/pods/5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056/volumes"
Jul 17 22:03:09.563255 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563169    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"]
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563517    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563527    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563537    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563542    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563558    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563563    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563577    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563582    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02934ee0-fb23-4c63-a8f8-b8e3adc0aa5f" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563662    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563669    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563676    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:09.563746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.563682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:09.568323 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.568304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:09.570931 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.570902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:03:09.574543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.574519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"]
Jul 17 22:03:09.682016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.681986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3857b2f-1340-422b-9991-787fbcf21bc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h\" (UID: \"a3857b2f-1340-422b-9991-787fbcf21bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:09.782535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.782478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3857b2f-1340-422b-9991-787fbcf21bc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h\" (UID: \"a3857b2f-1340-422b-9991-787fbcf21bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:09.783158 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.783136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3857b2f-1340-422b-9991-787fbcf21bc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h\" (UID: \"a3857b2f-1340-422b-9991-787fbcf21bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:09.880498 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:09.880416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:10.224278 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:10.224133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"]
Jul 17 22:03:10.227096 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:03:10.227059    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3857b2f_1340_422b_9991_787fbcf21bc4.slice/crio-bf2c70669d1457cdce3b72b65a0acdc1cc572d1e55f9615c2c4416e94de8782f WatchSource:0}: Error finding container bf2c70669d1457cdce3b72b65a0acdc1cc572d1e55f9615c2c4416e94de8782f: Status 404 returned error can't find the container with id bf2c70669d1457cdce3b72b65a0acdc1cc572d1e55f9615c2c4416e94de8782f
Jul 17 22:03:10.844139 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:10.844107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerStarted","Data":"a8aeb0788e5800bea545260d8a8564741926d918444f4f0fb55e3887c0f2262c"}
Jul 17 22:03:10.844541 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:10.844158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerStarted","Data":"bf2c70669d1457cdce3b72b65a0acdc1cc572d1e55f9615c2c4416e94de8782f"}
Jul 17 22:03:10.860835 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:10.860787    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" podStartSLOduration=1.860770627 podStartE2EDuration="1.860770627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:10.85890905 +0000 UTC m=+2154.771149339" watchObservedRunningTime="2026-07-17 22:03:10.860770627 +0000 UTC m=+2154.773010917"
Jul 17 22:03:11.850741 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:11.850709    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerID="a8aeb0788e5800bea545260d8a8564741926d918444f4f0fb55e3887c0f2262c" exitCode=6
Jul 17 22:03:11.851129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:11.850795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerDied","Data":"a8aeb0788e5800bea545260d8a8564741926d918444f4f0fb55e3887c0f2262c"}
Jul 17 22:03:11.851129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:11.850972    2563 scope.go:117] "RemoveContainer" containerID="a8aeb0788e5800bea545260d8a8564741926d918444f4f0fb55e3887c0f2262c"
Jul 17 22:03:12.857298 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:12.857267    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerID="bad38d44f93c76f5376c67a6a8052193b5c623f4529e4752d820332b41fc79c4" exitCode=6
Jul 17 22:03:12.857765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:12.857349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerDied","Data":"bad38d44f93c76f5376c67a6a8052193b5c623f4529e4752d820332b41fc79c4"}
Jul 17 22:03:12.857765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:12.857412    2563 scope.go:117] "RemoveContainer" containerID="a8aeb0788e5800bea545260d8a8564741926d918444f4f0fb55e3887c0f2262c"
Jul 17 22:03:12.857765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:12.857654    2563 scope.go:117] "RemoveContainer" containerID="bad38d44f93c76f5376c67a6a8052193b5c623f4529e4752d820332b41fc79c4"
Jul 17 22:03:12.857936 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:12.857913    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h_odh-ai-gateway-infra(a3857b2f-1340-422b-9991-787fbcf21bc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4"
Jul 17 22:03:13.864051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:13.864027    2563 scope.go:117] "RemoveContainer" containerID="bad38d44f93c76f5376c67a6a8052193b5c623f4529e4752d820332b41fc79c4"
Jul 17 22:03:13.864408 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:13.864261    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h_odh-ai-gateway-infra(a3857b2f-1340-422b-9991-787fbcf21bc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4"
Jul 17 22:03:17.415393 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415360    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"]
Jul 17 22:03:17.415785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:17.415785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415752    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:17.415785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:17.415785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415773    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:17.415924 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415853    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bcc07c6-d2a4-425d-bfe6-31f0c3ca0056" containerName="revoke-keys"
Jul 17 22:03:17.415924 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.415862    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84417881-2807-41d7-b21b-00e77fd2e481" containerName="revoke-keys"
Jul 17 22:03:17.420345 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.420326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:17.430138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.430114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"]
Jul 17 22:03:17.550623 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.550548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58a578e-5f99-47b3-8195-66cb53b4a3ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng\" (UID: \"a58a578e-5f99-47b3-8195-66cb53b4a3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:17.651539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.651491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58a578e-5f99-47b3-8195-66cb53b4a3ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng\" (UID: \"a58a578e-5f99-47b3-8195-66cb53b4a3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:17.652113 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.652091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58a578e-5f99-47b3-8195-66cb53b4a3ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng\" (UID: \"a58a578e-5f99-47b3-8195-66cb53b4a3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:17.730242 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.730172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:17.868161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.868131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"]
Jul 17 22:03:17.869195 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:03:17.869168    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58a578e_5f99_47b3_8195_66cb53b4a3ba.slice/crio-aad48afd2a65de31f2275ead4940b928e38f225351d104501b782fc3d9482140 WatchSource:0}: Error finding container aad48afd2a65de31f2275ead4940b928e38f225351d104501b782fc3d9482140: Status 404 returned error can't find the container with id aad48afd2a65de31f2275ead4940b928e38f225351d104501b782fc3d9482140
Jul 17 22:03:17.888211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:17.888167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" event={"ID":"a58a578e-5f99-47b3-8195-66cb53b4a3ba","Type":"ContainerStarted","Data":"aad48afd2a65de31f2275ead4940b928e38f225351d104501b782fc3d9482140"}
Jul 17 22:03:18.894215 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:18.894126    2563 generic.go:356] "Generic (PLEG): container finished" podID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerID="2415f987630d50db0dec62d4726e61e8c050118d847b5199607eb5e6ddfc517e" exitCode=6
Jul 17 22:03:18.894577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:18.894217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" event={"ID":"a58a578e-5f99-47b3-8195-66cb53b4a3ba","Type":"ContainerDied","Data":"2415f987630d50db0dec62d4726e61e8c050118d847b5199607eb5e6ddfc517e"}
Jul 17 22:03:18.894577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:18.894393    2563 scope.go:117] "RemoveContainer" containerID="2415f987630d50db0dec62d4726e61e8c050118d847b5199607eb5e6ddfc517e"
Jul 17 22:03:19.900301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:19.900207    2563 generic.go:356] "Generic (PLEG): container finished" podID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerID="4414bf63d765766261ce7beb63c10912a7ebba05f2d425cb413b4873eb3f8e78" exitCode=6
Jul 17 22:03:19.900301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:19.900264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" event={"ID":"a58a578e-5f99-47b3-8195-66cb53b4a3ba","Type":"ContainerDied","Data":"4414bf63d765766261ce7beb63c10912a7ebba05f2d425cb413b4873eb3f8e78"}
Jul 17 22:03:19.900757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:19.900334    2563 scope.go:117] "RemoveContainer" containerID="2415f987630d50db0dec62d4726e61e8c050118d847b5199607eb5e6ddfc517e"
Jul 17 22:03:19.900757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:19.900475    2563 scope.go:117] "RemoveContainer" containerID="4414bf63d765766261ce7beb63c10912a7ebba05f2d425cb413b4873eb3f8e78"
Jul 17 22:03:19.900757 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:19.900746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng_odh-ai-gateway-infra(a58a578e-5f99-47b3-8195-66cb53b4a3ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba"
Jul 17 22:03:20.907328 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:20.907301    2563 scope.go:117] "RemoveContainer" containerID="4414bf63d765766261ce7beb63c10912a7ebba05f2d425cb413b4873eb3f8e78"
Jul 17 22:03:20.907743 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:20.907495    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng_odh-ai-gateway-infra(a58a578e-5f99-47b3-8195-66cb53b4a3ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba"
Jul 17 22:03:25.244380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.244344    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"]
Jul 17 22:03:25.247742 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.247719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:25.257827 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.257803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"]
Jul 17 22:03:25.317180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.317143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837f9cbd-bdf1-405e-accd-cef86565d5f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g\" (UID: \"837f9cbd-bdf1-405e-accd-cef86565d5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:25.417680 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.417633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837f9cbd-bdf1-405e-accd-cef86565d5f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g\" (UID: \"837f9cbd-bdf1-405e-accd-cef86565d5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:25.418310 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.418281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837f9cbd-bdf1-405e-accd-cef86565d5f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g\" (UID: \"837f9cbd-bdf1-405e-accd-cef86565d5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:25.559201 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.559096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:25.696128 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.696102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"]
Jul 17 22:03:25.697913 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:03:25.697884    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837f9cbd_bdf1_405e_accd_cef86565d5f8.slice/crio-691a76f283c9aac72ab64662d741a4943f1b3c8290fa230a0e1b5fcb589e0da5 WatchSource:0}: Error finding container 691a76f283c9aac72ab64662d741a4943f1b3c8290fa230a0e1b5fcb589e0da5: Status 404 returned error can't find the container with id 691a76f283c9aac72ab64662d741a4943f1b3c8290fa230a0e1b5fcb589e0da5
Jul 17 22:03:25.932500 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.932455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerStarted","Data":"8b33956dbbd58f3b3710fa48957cc8aa8abc5bef69759bbe488513d22023cdc0"}
Jul 17 22:03:25.932662 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.932549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerStarted","Data":"691a76f283c9aac72ab64662d741a4943f1b3c8290fa230a0e1b5fcb589e0da5"}
Jul 17 22:03:25.948689 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:25.948640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" podStartSLOduration=0.948625219 podStartE2EDuration="948.625219ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:25.946575152 +0000 UTC m=+2169.858815445" watchObservedRunningTime="2026-07-17 22:03:25.948625219 +0000 UTC m=+2169.860865578"
Jul 17 22:03:26.939473 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:26.939438    2563 generic.go:356] "Generic (PLEG): container finished" podID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerID="8b33956dbbd58f3b3710fa48957cc8aa8abc5bef69759bbe488513d22023cdc0" exitCode=7
Jul 17 22:03:26.939970 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:26.939489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerDied","Data":"8b33956dbbd58f3b3710fa48957cc8aa8abc5bef69759bbe488513d22023cdc0"}
Jul 17 22:03:26.939970 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:26.939741    2563 scope.go:117] "RemoveContainer" containerID="8b33956dbbd58f3b3710fa48957cc8aa8abc5bef69759bbe488513d22023cdc0"
Jul 17 22:03:27.597103 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:27.597023    2563 scope.go:117] "RemoveContainer" containerID="bad38d44f93c76f5376c67a6a8052193b5c623f4529e4752d820332b41fc79c4"
Jul 17 22:03:27.946657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:27.946622    2563 generic.go:356] "Generic (PLEG): container finished" podID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerID="d6f602c87e89bf9ea616faeb32fe5bee7d19f67c141a690af82f583072e4b1a4" exitCode=7
Jul 17 22:03:27.947106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:27.946717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerDied","Data":"d6f602c87e89bf9ea616faeb32fe5bee7d19f67c141a690af82f583072e4b1a4"}
Jul 17 22:03:27.947106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:27.946787    2563 scope.go:117] "RemoveContainer" containerID="8b33956dbbd58f3b3710fa48957cc8aa8abc5bef69759bbe488513d22023cdc0"
Jul 17 22:03:27.947106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:27.946910    2563 scope.go:117] "RemoveContainer" containerID="d6f602c87e89bf9ea616faeb32fe5bee7d19f67c141a690af82f583072e4b1a4"
Jul 17 22:03:27.947295 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:27.947131    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g_odh-ai-gateway-infra(837f9cbd-bdf1-405e-accd-cef86565d5f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8"
Jul 17 22:03:27.948661 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:27.948631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerStarted","Data":"5004d67af74bf133a75f15a8378ab911bba15e51c7cfa5a697b4491452e00be8"}
Jul 17 22:03:28.955753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:28.955716    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerID="5004d67af74bf133a75f15a8378ab911bba15e51c7cfa5a697b4491452e00be8" exitCode=6
Jul 17 22:03:28.956187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:28.955783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerDied","Data":"5004d67af74bf133a75f15a8378ab911bba15e51c7cfa5a697b4491452e00be8"}
Jul 17 22:03:28.956187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:28.955841    2563 scope.go:117] "RemoveContainer" containerID="bad38d44f93c76f5376c67a6a8052193b5c623f4529e4752d820332b41fc79c4"
Jul 17 22:03:28.956187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:28.956143    2563 scope.go:117] "RemoveContainer" containerID="5004d67af74bf133a75f15a8378ab911bba15e51c7cfa5a697b4491452e00be8"
Jul 17 22:03:28.956363 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:28.956343    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h_odh-ai-gateway-infra(a3857b2f-1340-422b-9991-787fbcf21bc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4"
Jul 17 22:03:28.957709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:28.957690    2563 scope.go:117] "RemoveContainer" containerID="d6f602c87e89bf9ea616faeb32fe5bee7d19f67c141a690af82f583072e4b1a4"
Jul 17 22:03:28.957880 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:28.957863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g_odh-ai-gateway-infra(837f9cbd-bdf1-405e-accd-cef86565d5f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8"
Jul 17 22:03:28.983160 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:28.983135    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"]
Jul 17 22:03:30.102032 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.102009    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:30.150882 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.150849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3857b2f-1340-422b-9991-787fbcf21bc4-maas-api-service-ca\") pod \"a3857b2f-1340-422b-9991-787fbcf21bc4\" (UID: \"a3857b2f-1340-422b-9991-787fbcf21bc4\") "
Jul 17 22:03:30.151189 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.151165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3857b2f-1340-422b-9991-787fbcf21bc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3857b2f-1340-422b-9991-787fbcf21bc4" (UID: "a3857b2f-1340-422b-9991-787fbcf21bc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:30.251821 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.251734    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3857b2f-1340-422b-9991-787fbcf21bc4-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:03:30.970297 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.970268    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"
Jul 17 22:03:30.970471 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.970320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h" event={"ID":"a3857b2f-1340-422b-9991-787fbcf21bc4","Type":"ContainerDied","Data":"bf2c70669d1457cdce3b72b65a0acdc1cc572d1e55f9615c2c4416e94de8782f"}
Jul 17 22:03:30.970471 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.970406    2563 scope.go:117] "RemoveContainer" containerID="5004d67af74bf133a75f15a8378ab911bba15e51c7cfa5a697b4491452e00be8"
Jul 17 22:03:30.986515 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.986487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"]
Jul 17 22:03:30.992196 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:30.992174    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-mkt6h"]
Jul 17 22:03:32.602285 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:32.602243    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" path="/var/lib/kubelet/pods/a3857b2f-1340-422b-9991-787fbcf21bc4/volumes"
Jul 17 22:03:36.599699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:36.599668    2563 scope.go:117] "RemoveContainer" containerID="4414bf63d765766261ce7beb63c10912a7ebba05f2d425cb413b4873eb3f8e78"
Jul 17 22:03:37.001643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:37.001494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" event={"ID":"a58a578e-5f99-47b3-8195-66cb53b4a3ba","Type":"ContainerStarted","Data":"4111d4e1bd768e9923cd49a1df58c1a19508ce036852fe5ba68e2839c0a00429"}
Jul 17 22:03:37.020212 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:37.020162    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" podStartSLOduration=20.020146833 podStartE2EDuration="20.020146833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:37.018946843 +0000 UTC m=+2180.931187132" watchObservedRunningTime="2026-07-17 22:03:37.020146833 +0000 UTC m=+2180.932387122"
Jul 17 22:03:38.007786 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:38.007751    2563 generic.go:356] "Generic (PLEG): container finished" podID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerID="4111d4e1bd768e9923cd49a1df58c1a19508ce036852fe5ba68e2839c0a00429" exitCode=6
Jul 17 22:03:38.008164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:38.007825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" event={"ID":"a58a578e-5f99-47b3-8195-66cb53b4a3ba","Type":"ContainerDied","Data":"4111d4e1bd768e9923cd49a1df58c1a19508ce036852fe5ba68e2839c0a00429"}
Jul 17 22:03:38.008164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:38.007889    2563 scope.go:117] "RemoveContainer" containerID="4414bf63d765766261ce7beb63c10912a7ebba05f2d425cb413b4873eb3f8e78"
Jul 17 22:03:38.008291 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:38.008163    2563 scope.go:117] "RemoveContainer" containerID="4111d4e1bd768e9923cd49a1df58c1a19508ce036852fe5ba68e2839c0a00429"
Jul 17 22:03:38.008420 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:38.008395    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng_odh-ai-gateway-infra(a58a578e-5f99-47b3-8195-66cb53b4a3ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba"
Jul 17 22:03:38.031395 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:38.031366    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"]
Jul 17 22:03:39.157406 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:39.157375    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:39.231463 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:39.231434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58a578e-5f99-47b3-8195-66cb53b4a3ba-maas-api-service-ca\") pod \"a58a578e-5f99-47b3-8195-66cb53b4a3ba\" (UID: \"a58a578e-5f99-47b3-8195-66cb53b4a3ba\") "
Jul 17 22:03:39.231804 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:39.231783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58a578e-5f99-47b3-8195-66cb53b4a3ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a58a578e-5f99-47b3-8195-66cb53b4a3ba" (UID: "a58a578e-5f99-47b3-8195-66cb53b4a3ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:39.332583 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:39.332497    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58a578e-5f99-47b3-8195-66cb53b4a3ba-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:03:40.020227 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:40.020198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"
Jul 17 22:03:40.020227 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:40.020218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng" event={"ID":"a58a578e-5f99-47b3-8195-66cb53b4a3ba","Type":"ContainerDied","Data":"aad48afd2a65de31f2275ead4940b928e38f225351d104501b782fc3d9482140"}
Jul 17 22:03:40.020506 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:40.020292    2563 scope.go:117] "RemoveContainer" containerID="4111d4e1bd768e9923cd49a1df58c1a19508ce036852fe5ba68e2839c0a00429"
Jul 17 22:03:40.042320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:40.042294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"]
Jul 17 22:03:40.049735 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:40.049715    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-kllng"]
Jul 17 22:03:40.603861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:40.603831    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" path="/var/lib/kubelet/pods/a58a578e-5f99-47b3-8195-66cb53b4a3ba/volumes"
Jul 17 22:03:41.596782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:41.596745    2563 scope.go:117] "RemoveContainer" containerID="d6f602c87e89bf9ea616faeb32fe5bee7d19f67c141a690af82f583072e4b1a4"
Jul 17 22:03:42.031849 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:42.031817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerStarted","Data":"1adde703175463664d517b6ad398e6c356d4c6453c4c48d938198142c9dac6a3"}
Jul 17 22:03:43.038310 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:43.038272    2563 generic.go:356] "Generic (PLEG): container finished" podID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerID="1adde703175463664d517b6ad398e6c356d4c6453c4c48d938198142c9dac6a3" exitCode=7
Jul 17 22:03:43.038776 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:43.038329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerDied","Data":"1adde703175463664d517b6ad398e6c356d4c6453c4c48d938198142c9dac6a3"}
Jul 17 22:03:43.038776 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:43.038378    2563 scope.go:117] "RemoveContainer" containerID="d6f602c87e89bf9ea616faeb32fe5bee7d19f67c141a690af82f583072e4b1a4"
Jul 17 22:03:43.038776 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:43.038717    2563 scope.go:117] "RemoveContainer" containerID="1adde703175463664d517b6ad398e6c356d4c6453c4c48d938198142c9dac6a3"
Jul 17 22:03:43.038926 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:03:43.038900    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g_odh-ai-gateway-infra(837f9cbd-bdf1-405e-accd-cef86565d5f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8"
Jul 17 22:03:43.063290 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:43.063266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"]
Jul 17 22:03:44.173326 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:44.173303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:44.275763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:44.275731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837f9cbd-bdf1-405e-accd-cef86565d5f8-maas-api-service-ca\") pod \"837f9cbd-bdf1-405e-accd-cef86565d5f8\" (UID: \"837f9cbd-bdf1-405e-accd-cef86565d5f8\") "
Jul 17 22:03:44.276096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:44.276071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837f9cbd-bdf1-405e-accd-cef86565d5f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "837f9cbd-bdf1-405e-accd-cef86565d5f8" (UID: "837f9cbd-bdf1-405e-accd-cef86565d5f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:44.377109 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:44.377015    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837f9cbd-bdf1-405e-accd-cef86565d5f8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:03:45.051080 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:45.051049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"
Jul 17 22:03:45.051080 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:45.051060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g" event={"ID":"837f9cbd-bdf1-405e-accd-cef86565d5f8","Type":"ContainerDied","Data":"691a76f283c9aac72ab64662d741a4943f1b3c8290fa230a0e1b5fcb589e0da5"}
Jul 17 22:03:45.051338 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:45.051132    2563 scope.go:117] "RemoveContainer" containerID="1adde703175463664d517b6ad398e6c356d4c6453c4c48d938198142c9dac6a3"
Jul 17 22:03:45.068227 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:45.068201    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"]
Jul 17 22:03:45.072837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:45.072815    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wh47g"]
Jul 17 22:03:46.603217 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:03:46.603183    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" path="/var/lib/kubelet/pods/837f9cbd-bdf1-405e-accd-cef86565d5f8/volumes"
Jul 17 22:04:09.611947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.611911    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"]
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612238    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612258    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612268    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612274    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612284    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612289    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612295    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612299    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612306    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612316    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612323    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:09.612342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612337    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:09.612796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612400    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:09.612796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612409    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612414    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:09.612796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612421    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:09.612796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612428    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:09.612796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.612435    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:09.615334 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.615319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:09.617717 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.617694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:04:09.622333 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.622310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"]
Jul 17 22:04:09.803012 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.802975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e8b7f-367b-4fc2-8a1c-b7dbad378b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr\" (UID: \"068e8b7f-367b-4fc2-8a1c-b7dbad378b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:09.904511 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.904426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e8b7f-367b-4fc2-8a1c-b7dbad378b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr\" (UID: \"068e8b7f-367b-4fc2-8a1c-b7dbad378b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:09.905067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.905047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e8b7f-367b-4fc2-8a1c-b7dbad378b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr\" (UID: \"068e8b7f-367b-4fc2-8a1c-b7dbad378b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:09.926428 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:09.926406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:10.060940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:10.060916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"]
Jul 17 22:04:10.063137 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:04:10.063111    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068e8b7f_367b_4fc2_8a1c_b7dbad378b59.slice/crio-74b5e4c05cc32308ad7b70d194a5e8206f386f1c8e0306bedbbf30bf93dad02e WatchSource:0}: Error finding container 74b5e4c05cc32308ad7b70d194a5e8206f386f1c8e0306bedbbf30bf93dad02e: Status 404 returned error can't find the container with id 74b5e4c05cc32308ad7b70d194a5e8206f386f1c8e0306bedbbf30bf93dad02e
Jul 17 22:04:10.177806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:10.177772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" event={"ID":"068e8b7f-367b-4fc2-8a1c-b7dbad378b59","Type":"ContainerStarted","Data":"74b5e4c05cc32308ad7b70d194a5e8206f386f1c8e0306bedbbf30bf93dad02e"}
Jul 17 22:04:11.183755 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:11.183720    2563 generic.go:356] "Generic (PLEG): container finished" podID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerID="0246f0cc97a888ceb98c8713fff25d8a28c358942caca412e27a22e05d6a8141" exitCode=6
Jul 17 22:04:11.184172 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:11.183807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" event={"ID":"068e8b7f-367b-4fc2-8a1c-b7dbad378b59","Type":"ContainerDied","Data":"0246f0cc97a888ceb98c8713fff25d8a28c358942caca412e27a22e05d6a8141"}
Jul 17 22:04:11.184172 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:11.183947    2563 scope.go:117] "RemoveContainer" containerID="0246f0cc97a888ceb98c8713fff25d8a28c358942caca412e27a22e05d6a8141"
Jul 17 22:04:12.190142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:12.190110    2563 generic.go:356] "Generic (PLEG): container finished" podID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerID="6672de7a48c6217f9f35c062a3c2d5c5f06d850afb873e98a5d88df233f30dbb" exitCode=6
Jul 17 22:04:12.190650 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:12.190186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" event={"ID":"068e8b7f-367b-4fc2-8a1c-b7dbad378b59","Type":"ContainerDied","Data":"6672de7a48c6217f9f35c062a3c2d5c5f06d850afb873e98a5d88df233f30dbb"}
Jul 17 22:04:12.190650 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:12.190240    2563 scope.go:117] "RemoveContainer" containerID="0246f0cc97a888ceb98c8713fff25d8a28c358942caca412e27a22e05d6a8141"
Jul 17 22:04:12.190650 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:12.190484    2563 scope.go:117] "RemoveContainer" containerID="6672de7a48c6217f9f35c062a3c2d5c5f06d850afb873e98a5d88df233f30dbb"
Jul 17 22:04:12.190787 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:12.190765    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr_odh-ai-gateway-infra(068e8b7f-367b-4fc2-8a1c-b7dbad378b59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59"
Jul 17 22:04:13.196804 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:13.196779    2563 scope.go:117] "RemoveContainer" containerID="6672de7a48c6217f9f35c062a3c2d5c5f06d850afb873e98a5d88df233f30dbb"
Jul 17 22:04:13.197188 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:13.196988    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr_odh-ai-gateway-infra(068e8b7f-367b-4fc2-8a1c-b7dbad378b59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59"
Jul 17 22:04:17.447444 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.447402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"]
Jul 17 22:04:17.447977 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.447957    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:17.448061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.447980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:17.448061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.448000    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:17.448061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.448010    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:17.448208 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.448106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="837f9cbd-bdf1-405e-accd-cef86565d5f8" containerName="revoke-keys"
Jul 17 22:04:17.448208 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.448122    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58a578e-5f99-47b3-8195-66cb53b4a3ba" containerName="revoke-keys"
Jul 17 22:04:17.448208 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.448133    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3857b2f-1340-422b-9991-787fbcf21bc4" containerName="revoke-keys"
Jul 17 22:04:17.452494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.452471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:17.457689 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.457662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"]
Jul 17 22:04:17.472974 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.472944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da3d340-c1a2-4140-a072-daa3a03b3a9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l\" (UID: \"2da3d340-c1a2-4140-a072-daa3a03b3a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:17.573767 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.573675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da3d340-c1a2-4140-a072-daa3a03b3a9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l\" (UID: \"2da3d340-c1a2-4140-a072-daa3a03b3a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:17.574315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.574292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da3d340-c1a2-4140-a072-daa3a03b3a9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l\" (UID: \"2da3d340-c1a2-4140-a072-daa3a03b3a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:17.765673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.765640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:17.899397 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:17.899299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"]
Jul 17 22:04:17.901721 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:04:17.901688    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da3d340_c1a2_4140_a072_daa3a03b3a9c.slice/crio-5c9e19a0c8b0e2519b0e63d367bb9894e413ee660bf697dfec4ba89f2866f2e9 WatchSource:0}: Error finding container 5c9e19a0c8b0e2519b0e63d367bb9894e413ee660bf697dfec4ba89f2866f2e9: Status 404 returned error can't find the container with id 5c9e19a0c8b0e2519b0e63d367bb9894e413ee660bf697dfec4ba89f2866f2e9
Jul 17 22:04:18.222306 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:18.222274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" event={"ID":"2da3d340-c1a2-4140-a072-daa3a03b3a9c","Type":"ContainerStarted","Data":"58e98eb4f22a7d1cfb1d3a3d9ed8c153641039c14938b4fd86dddb8c5bde10c2"}
Jul 17 22:04:18.222473 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:18.222324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" event={"ID":"2da3d340-c1a2-4140-a072-daa3a03b3a9c","Type":"ContainerStarted","Data":"5c9e19a0c8b0e2519b0e63d367bb9894e413ee660bf697dfec4ba89f2866f2e9"}
Jul 17 22:04:18.239994 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:18.239944    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" podStartSLOduration=1.2399304 podStartE2EDuration="1.2399304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:04:18.238296025 +0000 UTC m=+2222.150536316" watchObservedRunningTime="2026-07-17 22:04:18.2399304 +0000 UTC m=+2222.152170688"
Jul 17 22:04:19.228613 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:19.228567    2563 generic.go:356] "Generic (PLEG): container finished" podID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerID="58e98eb4f22a7d1cfb1d3a3d9ed8c153641039c14938b4fd86dddb8c5bde10c2" exitCode=6
Jul 17 22:04:19.229040 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:19.228661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" event={"ID":"2da3d340-c1a2-4140-a072-daa3a03b3a9c","Type":"ContainerDied","Data":"58e98eb4f22a7d1cfb1d3a3d9ed8c153641039c14938b4fd86dddb8c5bde10c2"}
Jul 17 22:04:19.229040 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:19.228822    2563 scope.go:117] "RemoveContainer" containerID="58e98eb4f22a7d1cfb1d3a3d9ed8c153641039c14938b4fd86dddb8c5bde10c2"
Jul 17 22:04:20.235203 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:20.235171    2563 generic.go:356] "Generic (PLEG): container finished" podID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerID="491ae94d1e2788b20cebc30d2a70f1c1a99172b2e6989072193eb69c2e936312" exitCode=6
Jul 17 22:04:20.235687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:20.235258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" event={"ID":"2da3d340-c1a2-4140-a072-daa3a03b3a9c","Type":"ContainerDied","Data":"491ae94d1e2788b20cebc30d2a70f1c1a99172b2e6989072193eb69c2e936312"}
Jul 17 22:04:20.235687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:20.235321    2563 scope.go:117] "RemoveContainer" containerID="58e98eb4f22a7d1cfb1d3a3d9ed8c153641039c14938b4fd86dddb8c5bde10c2"
Jul 17 22:04:20.235687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:20.235522    2563 scope.go:117] "RemoveContainer" containerID="491ae94d1e2788b20cebc30d2a70f1c1a99172b2e6989072193eb69c2e936312"
Jul 17 22:04:20.236041 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:20.235764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l_odh-ai-gateway-infra(2da3d340-c1a2-4140-a072-daa3a03b3a9c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c"
Jul 17 22:04:21.248808 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:21.248780    2563 scope.go:117] "RemoveContainer" containerID="491ae94d1e2788b20cebc30d2a70f1c1a99172b2e6989072193eb69c2e936312"
Jul 17 22:04:21.249185 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:21.248967    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l_odh-ai-gateway-infra(2da3d340-c1a2-4140-a072-daa3a03b3a9c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c"
Jul 17 22:04:22.805613 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:22.805565    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"]
Jul 17 22:04:22.809025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:22.809007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:22.815692 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:22.815660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e01fc8-df4a-4184-8c1a-b23a909b7f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4\" (UID: \"29e01fc8-df4a-4184-8c1a-b23a909b7f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:22.816026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:22.816002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"]
Jul 17 22:04:22.916674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:22.916642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e01fc8-df4a-4184-8c1a-b23a909b7f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4\" (UID: \"29e01fc8-df4a-4184-8c1a-b23a909b7f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:22.917259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:22.917237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e01fc8-df4a-4184-8c1a-b23a909b7f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4\" (UID: \"29e01fc8-df4a-4184-8c1a-b23a909b7f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:23.121504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:23.121422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:23.257843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:23.257810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"]
Jul 17 22:04:23.261178 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:04:23.261150    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e01fc8_df4a_4184_8c1a_b23a909b7f9f.slice/crio-c251f6135fcbe9872c7bc5abbf62fe9deb18fb2bcbdebddf2216328a17f7b59d WatchSource:0}: Error finding container c251f6135fcbe9872c7bc5abbf62fe9deb18fb2bcbdebddf2216328a17f7b59d: Status 404 returned error can't find the container with id c251f6135fcbe9872c7bc5abbf62fe9deb18fb2bcbdebddf2216328a17f7b59d
Jul 17 22:04:24.264467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:24.264434    2563 generic.go:356] "Generic (PLEG): container finished" podID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerID="728fdb727954b87986052dddbad69d9bc6415454d428a39ebda9c8192389b400" exitCode=7
Jul 17 22:04:24.264925 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:24.264525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" event={"ID":"29e01fc8-df4a-4184-8c1a-b23a909b7f9f","Type":"ContainerDied","Data":"728fdb727954b87986052dddbad69d9bc6415454d428a39ebda9c8192389b400"}
Jul 17 22:04:24.264925 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:24.264584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" event={"ID":"29e01fc8-df4a-4184-8c1a-b23a909b7f9f","Type":"ContainerStarted","Data":"c251f6135fcbe9872c7bc5abbf62fe9deb18fb2bcbdebddf2216328a17f7b59d"}
Jul 17 22:04:24.264925 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:24.264711    2563 scope.go:117] "RemoveContainer" containerID="728fdb727954b87986052dddbad69d9bc6415454d428a39ebda9c8192389b400"
Jul 17 22:04:25.271237 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:25.271204    2563 generic.go:356] "Generic (PLEG): container finished" podID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerID="147688ab50a50d96102e6977810a3fd8a3058efe7f3ba2ce9a9e3d77f4110cf7" exitCode=7
Jul 17 22:04:25.271705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:25.271267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" event={"ID":"29e01fc8-df4a-4184-8c1a-b23a909b7f9f","Type":"ContainerDied","Data":"147688ab50a50d96102e6977810a3fd8a3058efe7f3ba2ce9a9e3d77f4110cf7"}
Jul 17 22:04:25.271705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:25.271328    2563 scope.go:117] "RemoveContainer" containerID="728fdb727954b87986052dddbad69d9bc6415454d428a39ebda9c8192389b400"
Jul 17 22:04:25.271705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:25.271530    2563 scope.go:117] "RemoveContainer" containerID="147688ab50a50d96102e6977810a3fd8a3058efe7f3ba2ce9a9e3d77f4110cf7"
Jul 17 22:04:25.271821 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:25.271790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4_odh-ai-gateway-infra(29e01fc8-df4a-4184-8c1a-b23a909b7f9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f"
Jul 17 22:04:26.278175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:26.278141    2563 scope.go:117] "RemoveContainer" containerID="147688ab50a50d96102e6977810a3fd8a3058efe7f3ba2ce9a9e3d77f4110cf7"
Jul 17 22:04:26.278564 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:26.278351    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4_odh-ai-gateway-infra(29e01fc8-df4a-4184-8c1a-b23a909b7f9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f"
Jul 17 22:04:26.599133 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:26.599027    2563 scope.go:117] "RemoveContainer" containerID="6672de7a48c6217f9f35c062a3c2d5c5f06d850afb873e98a5d88df233f30dbb"
Jul 17 22:04:27.284438 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:27.284403    2563 generic.go:356] "Generic (PLEG): container finished" podID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerID="051874a4463ce115974696b58f538db4a560a4b11bba98e61955ffc4095dc6d7" exitCode=6
Jul 17 22:04:27.284928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:27.284458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" event={"ID":"068e8b7f-367b-4fc2-8a1c-b7dbad378b59","Type":"ContainerDied","Data":"051874a4463ce115974696b58f538db4a560a4b11bba98e61955ffc4095dc6d7"}
Jul 17 22:04:27.284928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:27.284520    2563 scope.go:117] "RemoveContainer" containerID="6672de7a48c6217f9f35c062a3c2d5c5f06d850afb873e98a5d88df233f30dbb"
Jul 17 22:04:27.284928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:27.284866    2563 scope.go:117] "RemoveContainer" containerID="051874a4463ce115974696b58f538db4a560a4b11bba98e61955ffc4095dc6d7"
Jul 17 22:04:27.285103 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:27.285080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr_odh-ai-gateway-infra(068e8b7f-367b-4fc2-8a1c-b7dbad378b59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59"
Jul 17 22:04:28.310695 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:28.310664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"]
Jul 17 22:04:28.449539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:28.449516    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:28.463112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:28.463089    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e8b7f-367b-4fc2-8a1c-b7dbad378b59-maas-api-service-ca\") pod \"068e8b7f-367b-4fc2-8a1c-b7dbad378b59\" (UID: \"068e8b7f-367b-4fc2-8a1c-b7dbad378b59\") "
Jul 17 22:04:28.463400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:28.463380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068e8b7f-367b-4fc2-8a1c-b7dbad378b59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "068e8b7f-367b-4fc2-8a1c-b7dbad378b59" (UID: "068e8b7f-367b-4fc2-8a1c-b7dbad378b59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:28.564648 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:28.564536    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e8b7f-367b-4fc2-8a1c-b7dbad378b59-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:04:29.297013 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:29.296984    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"
Jul 17 22:04:29.297196 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:29.297041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr" event={"ID":"068e8b7f-367b-4fc2-8a1c-b7dbad378b59","Type":"ContainerDied","Data":"74b5e4c05cc32308ad7b70d194a5e8206f386f1c8e0306bedbbf30bf93dad02e"}
Jul 17 22:04:29.297196 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:29.297097    2563 scope.go:117] "RemoveContainer" containerID="051874a4463ce115974696b58f538db4a560a4b11bba98e61955ffc4095dc6d7"
Jul 17 22:04:29.338201 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:29.338172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"]
Jul 17 22:04:29.344347 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:29.344327    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-flvvr"]
Jul 17 22:04:30.608816 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:30.608783    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" path="/var/lib/kubelet/pods/068e8b7f-367b-4fc2-8a1c-b7dbad378b59/volumes"
Jul 17 22:04:35.596761 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:35.596732    2563 scope.go:117] "RemoveContainer" containerID="491ae94d1e2788b20cebc30d2a70f1c1a99172b2e6989072193eb69c2e936312"
Jul 17 22:04:36.333344 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:36.333312    2563 generic.go:356] "Generic (PLEG): container finished" podID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerID="daeba7598694d2edf90e0287be438a741deddc44bd7e122ee4b75c4343b85c8b" exitCode=6
Jul 17 22:04:36.333520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:36.333387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" event={"ID":"2da3d340-c1a2-4140-a072-daa3a03b3a9c","Type":"ContainerDied","Data":"daeba7598694d2edf90e0287be438a741deddc44bd7e122ee4b75c4343b85c8b"}
Jul 17 22:04:36.333520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:36.333463    2563 scope.go:117] "RemoveContainer" containerID="491ae94d1e2788b20cebc30d2a70f1c1a99172b2e6989072193eb69c2e936312"
Jul 17 22:04:36.333838 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:36.333819    2563 scope.go:117] "RemoveContainer" containerID="daeba7598694d2edf90e0287be438a741deddc44bd7e122ee4b75c4343b85c8b"
Jul 17 22:04:36.334081 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:36.334062    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l_odh-ai-gateway-infra(2da3d340-c1a2-4140-a072-daa3a03b3a9c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c"
Jul 17 22:04:37.359925 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:37.359887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"]
Jul 17 22:04:37.505801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:37.505778    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:37.540402 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:37.540367    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da3d340-c1a2-4140-a072-daa3a03b3a9c-maas-api-service-ca\") pod \"2da3d340-c1a2-4140-a072-daa3a03b3a9c\" (UID: \"2da3d340-c1a2-4140-a072-daa3a03b3a9c\") "
Jul 17 22:04:37.540764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:37.540737    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da3d340-c1a2-4140-a072-daa3a03b3a9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2da3d340-c1a2-4140-a072-daa3a03b3a9c" (UID: "2da3d340-c1a2-4140-a072-daa3a03b3a9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:37.641365 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:37.641280    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da3d340-c1a2-4140-a072-daa3a03b3a9c-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:04:38.346890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.346862    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"
Jul 17 22:04:38.346890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.346866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l" event={"ID":"2da3d340-c1a2-4140-a072-daa3a03b3a9c","Type":"ContainerDied","Data":"5c9e19a0c8b0e2519b0e63d367bb9894e413ee660bf697dfec4ba89f2866f2e9"}
Jul 17 22:04:38.347078 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.346929    2563 scope.go:117] "RemoveContainer" containerID="daeba7598694d2edf90e0287be438a741deddc44bd7e122ee4b75c4343b85c8b"
Jul 17 22:04:38.373684 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.373654    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"]
Jul 17 22:04:38.375994 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.375971    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-4sn9l"]
Jul 17 22:04:38.596809 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.596778    2563 scope.go:117] "RemoveContainer" containerID="147688ab50a50d96102e6977810a3fd8a3058efe7f3ba2ce9a9e3d77f4110cf7"
Jul 17 22:04:38.602745 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:38.602716    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" path="/var/lib/kubelet/pods/2da3d340-c1a2-4140-a072-daa3a03b3a9c/volumes"
Jul 17 22:04:39.353478 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:39.353438    2563 generic.go:356] "Generic (PLEG): container finished" podID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerID="a4c0588a0683da0d0c64aa98c973930210295466243bad964b4c49ea1d1f84ab" exitCode=7
Jul 17 22:04:39.353659 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:39.353497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" event={"ID":"29e01fc8-df4a-4184-8c1a-b23a909b7f9f","Type":"ContainerDied","Data":"a4c0588a0683da0d0c64aa98c973930210295466243bad964b4c49ea1d1f84ab"}
Jul 17 22:04:39.353659 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:39.353567    2563 scope.go:117] "RemoveContainer" containerID="147688ab50a50d96102e6977810a3fd8a3058efe7f3ba2ce9a9e3d77f4110cf7"
Jul 17 22:04:39.353923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:39.353901    2563 scope.go:117] "RemoveContainer" containerID="a4c0588a0683da0d0c64aa98c973930210295466243bad964b4c49ea1d1f84ab"
Jul 17 22:04:39.354149 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:04:39.354130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4_odh-ai-gateway-infra(29e01fc8-df4a-4184-8c1a-b23a909b7f9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f"
Jul 17 22:04:40.379738 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:40.379702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"]
Jul 17 22:04:40.518311 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:40.518284    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:40.566720 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:40.566685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e01fc8-df4a-4184-8c1a-b23a909b7f9f-maas-api-service-ca\") pod \"29e01fc8-df4a-4184-8c1a-b23a909b7f9f\" (UID: \"29e01fc8-df4a-4184-8c1a-b23a909b7f9f\") "
Jul 17 22:04:40.567136 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:40.567108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e01fc8-df4a-4184-8c1a-b23a909b7f9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29e01fc8-df4a-4184-8c1a-b23a909b7f9f" (UID: "29e01fc8-df4a-4184-8c1a-b23a909b7f9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:40.667701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:40.667671    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e01fc8-df4a-4184-8c1a-b23a909b7f9f-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:04:41.366796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:41.366760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4" event={"ID":"29e01fc8-df4a-4184-8c1a-b23a909b7f9f","Type":"ContainerDied","Data":"c251f6135fcbe9872c7bc5abbf62fe9deb18fb2bcbdebddf2216328a17f7b59d"}
Jul 17 22:04:41.366956 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:41.366820    2563 scope.go:117] "RemoveContainer" containerID="a4c0588a0683da0d0c64aa98c973930210295466243bad964b4c49ea1d1f84ab"
Jul 17 22:04:41.366956 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:41.366779    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"
Jul 17 22:04:41.382142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:41.382118    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"]
Jul 17 22:04:41.387944 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:41.387921    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cfwn4"]
Jul 17 22:04:42.602359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:42.602327    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" path="/var/lib/kubelet/pods/29e01fc8-df4a-4184-8c1a-b23a909b7f9f/volumes"
Jul 17 22:04:59.655653 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.655616    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"]
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.655996    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656017    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656031    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656048    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656053    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656059    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656064    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656074    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656079    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656090    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656095    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656161    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656171    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656177    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.656757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656183    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.656757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656189    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:04:59.656757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:04:59.656757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.656206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:04:59.659129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.659114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:04:59.661532 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.661512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:04:59.665869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.665844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"]
Jul 17 22:04:59.832758 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.832724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08419fb-c3f3-40a4-8edc-893dd5cab7c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7\" (UID: \"f08419fb-c3f3-40a4-8edc-893dd5cab7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:04:59.933755 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.933724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08419fb-c3f3-40a4-8edc-893dd5cab7c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7\" (UID: \"f08419fb-c3f3-40a4-8edc-893dd5cab7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:04:59.934342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.934321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08419fb-c3f3-40a4-8edc-893dd5cab7c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7\" (UID: \"f08419fb-c3f3-40a4-8edc-893dd5cab7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:04:59.971061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:04:59.971033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:05:00.106419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:00.106393    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"]
Jul 17 22:05:00.108643 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:05:00.108610    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08419fb_c3f3_40a4_8edc_893dd5cab7c8.slice/crio-44d57836a5e37e967b5dae95e0554bfaa6d93249a0985be6992f3125b95d73b9 WatchSource:0}: Error finding container 44d57836a5e37e967b5dae95e0554bfaa6d93249a0985be6992f3125b95d73b9: Status 404 returned error can't find the container with id 44d57836a5e37e967b5dae95e0554bfaa6d93249a0985be6992f3125b95d73b9
Jul 17 22:05:00.462502 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:00.462465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" event={"ID":"f08419fb-c3f3-40a4-8edc-893dd5cab7c8","Type":"ContainerStarted","Data":"8e92e5c269a9b51b071e103d2a97a7ecb430e02f79f8eeeb57f030b816ccaa2c"}
Jul 17 22:05:00.462684 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:00.462517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" event={"ID":"f08419fb-c3f3-40a4-8edc-893dd5cab7c8","Type":"ContainerStarted","Data":"44d57836a5e37e967b5dae95e0554bfaa6d93249a0985be6992f3125b95d73b9"}
Jul 17 22:05:00.479679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:00.479630    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" podStartSLOduration=1.479615356 podStartE2EDuration="1.479615356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:00.478128209 +0000 UTC m=+2264.390368497" watchObservedRunningTime="2026-07-17 22:05:00.479615356 +0000 UTC m=+2264.391855635"
Jul 17 22:05:01.468908 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:01.468875    2563 generic.go:356] "Generic (PLEG): container finished" podID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerID="8e92e5c269a9b51b071e103d2a97a7ecb430e02f79f8eeeb57f030b816ccaa2c" exitCode=6
Jul 17 22:05:01.469319 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:01.468960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" event={"ID":"f08419fb-c3f3-40a4-8edc-893dd5cab7c8","Type":"ContainerDied","Data":"8e92e5c269a9b51b071e103d2a97a7ecb430e02f79f8eeeb57f030b816ccaa2c"}
Jul 17 22:05:01.469319 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:01.469122    2563 scope.go:117] "RemoveContainer" containerID="8e92e5c269a9b51b071e103d2a97a7ecb430e02f79f8eeeb57f030b816ccaa2c"
Jul 17 22:05:02.479915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:02.479882    2563 generic.go:356] "Generic (PLEG): container finished" podID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerID="7fdddecd5927fe12bf33e59d58549ad9cce0fdd8e0aa155675e5431b40951e02" exitCode=6
Jul 17 22:05:02.480315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:02.479947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" event={"ID":"f08419fb-c3f3-40a4-8edc-893dd5cab7c8","Type":"ContainerDied","Data":"7fdddecd5927fe12bf33e59d58549ad9cce0fdd8e0aa155675e5431b40951e02"}
Jul 17 22:05:02.480315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:02.480019    2563 scope.go:117] "RemoveContainer" containerID="8e92e5c269a9b51b071e103d2a97a7ecb430e02f79f8eeeb57f030b816ccaa2c"
Jul 17 22:05:02.480315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:02.480195    2563 scope.go:117] "RemoveContainer" containerID="7fdddecd5927fe12bf33e59d58549ad9cce0fdd8e0aa155675e5431b40951e02"
Jul 17 22:05:02.480506 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:02.480428    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7_odh-ai-gateway-infra(f08419fb-c3f3-40a4-8edc-893dd5cab7c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8"
Jul 17 22:05:03.486853 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:03.486820    2563 scope.go:117] "RemoveContainer" containerID="7fdddecd5927fe12bf33e59d58549ad9cce0fdd8e0aa155675e5431b40951e02"
Jul 17 22:05:03.487321 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:03.487061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7_odh-ai-gateway-infra(f08419fb-c3f3-40a4-8edc-893dd5cab7c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8"
Jul 17 22:05:12.849351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.849313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"]
Jul 17 22:05:12.849878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.849697    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:05:12.849878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.849709    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="068e8b7f-367b-4fc2-8a1c-b7dbad378b59" containerName="revoke-keys"
Jul 17 22:05:12.849878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.849719    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:05:12.849878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.849725    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da3d340-c1a2-4140-a072-daa3a03b3a9c" containerName="revoke-keys"
Jul 17 22:05:12.849878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.849782    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e01fc8-df4a-4184-8c1a-b23a909b7f9f" containerName="revoke-keys"
Jul 17 22:05:12.854252 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.854231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:12.859538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.859515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"]
Jul 17 22:05:12.952649 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:12.952579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f65809a-152f-453b-897c-0fc4d6b06e9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-x255s\" (UID: \"7f65809a-152f-453b-897c-0fc4d6b06e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:13.053156 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:13.053115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f65809a-152f-453b-897c-0fc4d6b06e9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-x255s\" (UID: \"7f65809a-152f-453b-897c-0fc4d6b06e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:13.053757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:13.053738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f65809a-152f-453b-897c-0fc4d6b06e9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-x255s\" (UID: \"7f65809a-152f-453b-897c-0fc4d6b06e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:13.167128 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:13.167098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:13.308259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:13.308217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"]
Jul 17 22:05:13.310261 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:05:13.310222    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f65809a_152f_453b_897c_0fc4d6b06e9f.slice/crio-c14582c4cd8aff009283c7cc3ff267520c7f0406d74ab7ede849f34bcf1d7401 WatchSource:0}: Error finding container c14582c4cd8aff009283c7cc3ff267520c7f0406d74ab7ede849f34bcf1d7401: Status 404 returned error can't find the container with id c14582c4cd8aff009283c7cc3ff267520c7f0406d74ab7ede849f34bcf1d7401
Jul 17 22:05:13.537180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:13.537143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" event={"ID":"7f65809a-152f-453b-897c-0fc4d6b06e9f","Type":"ContainerStarted","Data":"c14582c4cd8aff009283c7cc3ff267520c7f0406d74ab7ede849f34bcf1d7401"}
Jul 17 22:05:14.543183 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:14.543149    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerID="6f5f6c687281c83ff47b13ffbfc13ab90f9bfb19f2a888b524bcd60bc7b4bb35" exitCode=7
Jul 17 22:05:14.543592 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:14.543236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" event={"ID":"7f65809a-152f-453b-897c-0fc4d6b06e9f","Type":"ContainerDied","Data":"6f5f6c687281c83ff47b13ffbfc13ab90f9bfb19f2a888b524bcd60bc7b4bb35"}
Jul 17 22:05:14.543592 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:14.543419    2563 scope.go:117] "RemoveContainer" containerID="6f5f6c687281c83ff47b13ffbfc13ab90f9bfb19f2a888b524bcd60bc7b4bb35"
Jul 17 22:05:15.331441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.331402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"]
Jul 17 22:05:15.334801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.334780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:15.342730 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.342706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"]
Jul 17 22:05:15.370716 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.370688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac73474-55e3-4852-a306-86c9e1d1e13e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2\" (UID: \"bac73474-55e3-4852-a306-86c9e1d1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:15.471260 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.471228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac73474-55e3-4852-a306-86c9e1d1e13e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2\" (UID: \"bac73474-55e3-4852-a306-86c9e1d1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:15.471861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.471843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac73474-55e3-4852-a306-86c9e1d1e13e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2\" (UID: \"bac73474-55e3-4852-a306-86c9e1d1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:15.550078 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.550048    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerID="9809ff792ad9726d4360be8a29e56097ba03c31ac49e7e49b79aab9123bd83eb" exitCode=7
Jul 17 22:05:15.550462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.550133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" event={"ID":"7f65809a-152f-453b-897c-0fc4d6b06e9f","Type":"ContainerDied","Data":"9809ff792ad9726d4360be8a29e56097ba03c31ac49e7e49b79aab9123bd83eb"}
Jul 17 22:05:15.550462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.550192    2563 scope.go:117] "RemoveContainer" containerID="6f5f6c687281c83ff47b13ffbfc13ab90f9bfb19f2a888b524bcd60bc7b4bb35"
Jul 17 22:05:15.550462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.550417    2563 scope.go:117] "RemoveContainer" containerID="9809ff792ad9726d4360be8a29e56097ba03c31ac49e7e49b79aab9123bd83eb"
Jul 17 22:05:15.550673 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:15.550655    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-x255s_odh-ai-gateway-infra(7f65809a-152f-453b-897c-0fc4d6b06e9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f"
Jul 17 22:05:15.646130 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.646092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:15.784382 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:15.784358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"]
Jul 17 22:05:15.786829 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:05:15.786800    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac73474_55e3_4852_a306_86c9e1d1e13e.slice/crio-d31bd4d1b87e06feaa2a608a6d5c8e29d12b0b45f858515234b92cccec1a53a4 WatchSource:0}: Error finding container d31bd4d1b87e06feaa2a608a6d5c8e29d12b0b45f858515234b92cccec1a53a4: Status 404 returned error can't find the container with id d31bd4d1b87e06feaa2a608a6d5c8e29d12b0b45f858515234b92cccec1a53a4
Jul 17 22:05:16.556056 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:16.556024    2563 generic.go:356] "Generic (PLEG): container finished" podID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerID="26eeac9cbf1ace1611c3338f05f1a5f828a7aaf0940f2affaebcab1d09cbba15" exitCode=6
Jul 17 22:05:16.556519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:16.556101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" event={"ID":"bac73474-55e3-4852-a306-86c9e1d1e13e","Type":"ContainerDied","Data":"26eeac9cbf1ace1611c3338f05f1a5f828a7aaf0940f2affaebcab1d09cbba15"}
Jul 17 22:05:16.556519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:16.556155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" event={"ID":"bac73474-55e3-4852-a306-86c9e1d1e13e","Type":"ContainerStarted","Data":"d31bd4d1b87e06feaa2a608a6d5c8e29d12b0b45f858515234b92cccec1a53a4"}
Jul 17 22:05:16.556519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:16.556289    2563 scope.go:117] "RemoveContainer" containerID="26eeac9cbf1ace1611c3338f05f1a5f828a7aaf0940f2affaebcab1d09cbba15"
Jul 17 22:05:16.557946 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:16.557931    2563 scope.go:117] "RemoveContainer" containerID="9809ff792ad9726d4360be8a29e56097ba03c31ac49e7e49b79aab9123bd83eb"
Jul 17 22:05:16.558090 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:16.558072    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-x255s_odh-ai-gateway-infra(7f65809a-152f-453b-897c-0fc4d6b06e9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f"
Jul 17 22:05:16.598875 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:16.598824    2563 scope.go:117] "RemoveContainer" containerID="7fdddecd5927fe12bf33e59d58549ad9cce0fdd8e0aa155675e5431b40951e02"
Jul 17 22:05:17.488770 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.488731    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"]
Jul 17 22:05:17.492380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.492362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:17.500124 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.500094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"]
Jul 17 22:05:17.564719 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.564684    2563 generic.go:356] "Generic (PLEG): container finished" podID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerID="52e40be30e9395aff9f7f08ac2a4a4d3e8cb38e22006c89f40c96e61c32a667b" exitCode=6
Jul 17 22:05:17.565170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.564756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" event={"ID":"f08419fb-c3f3-40a4-8edc-893dd5cab7c8","Type":"ContainerDied","Data":"52e40be30e9395aff9f7f08ac2a4a4d3e8cb38e22006c89f40c96e61c32a667b"}
Jul 17 22:05:17.565170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.564819    2563 scope.go:117] "RemoveContainer" containerID="7fdddecd5927fe12bf33e59d58549ad9cce0fdd8e0aa155675e5431b40951e02"
Jul 17 22:05:17.565170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.565169    2563 scope.go:117] "RemoveContainer" containerID="52e40be30e9395aff9f7f08ac2a4a4d3e8cb38e22006c89f40c96e61c32a667b"
Jul 17 22:05:17.565415 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:17.565392    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7_odh-ai-gateway-infra(f08419fb-c3f3-40a4-8edc-893dd5cab7c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8"
Jul 17 22:05:17.566810 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.566791    2563 generic.go:356] "Generic (PLEG): container finished" podID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerID="ebe33c30714623993dac85ffb44ca87576d01f7e1563f402a1e06de7b137ba88" exitCode=6
Jul 17 22:05:17.566901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.566864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" event={"ID":"bac73474-55e3-4852-a306-86c9e1d1e13e","Type":"ContainerDied","Data":"ebe33c30714623993dac85ffb44ca87576d01f7e1563f402a1e06de7b137ba88"}
Jul 17 22:05:17.567042 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.567029    2563 scope.go:117] "RemoveContainer" containerID="ebe33c30714623993dac85ffb44ca87576d01f7e1563f402a1e06de7b137ba88"
Jul 17 22:05:17.567198 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:17.567183    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2_odh-ai-gateway-infra(bac73474-55e3-4852-a306-86c9e1d1e13e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e"
Jul 17 22:05:17.576536 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.576510    2563 scope.go:117] "RemoveContainer" containerID="26eeac9cbf1ace1611c3338f05f1a5f828a7aaf0940f2affaebcab1d09cbba15"
Jul 17 22:05:17.587071 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.587043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd\" (UID: \"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:17.687777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.687743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd\" (UID: \"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:17.688351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.688331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd\" (UID: \"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:17.804475 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.804393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:17.945774 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:17.945622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"]
Jul 17 22:05:17.948672 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:05:17.948639    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83308f8_00cd_46e6_a3bc_d9c13b6ac4d3.slice/crio-b73eae9ec274ef31e66d18e20e9c43c9b9350ad60d993e4ec3e71dc15f23c40b WatchSource:0}: Error finding container b73eae9ec274ef31e66d18e20e9c43c9b9350ad60d993e4ec3e71dc15f23c40b: Status 404 returned error can't find the container with id b73eae9ec274ef31e66d18e20e9c43c9b9350ad60d993e4ec3e71dc15f23c40b
Jul 17 22:05:18.574400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.574371    2563 generic.go:356] "Generic (PLEG): container finished" podID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerID="69cd65f3004ac0a494c742a2eb5aa02ad6d3dae9155154622e14e6cfcbe1cf78" exitCode=6
Jul 17 22:05:18.574779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.574451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" event={"ID":"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3","Type":"ContainerDied","Data":"69cd65f3004ac0a494c742a2eb5aa02ad6d3dae9155154622e14e6cfcbe1cf78"}
Jul 17 22:05:18.574779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.574508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" event={"ID":"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3","Type":"ContainerStarted","Data":"b73eae9ec274ef31e66d18e20e9c43c9b9350ad60d993e4ec3e71dc15f23c40b"}
Jul 17 22:05:18.574779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.574583    2563 scope.go:117] "RemoveContainer" containerID="69cd65f3004ac0a494c742a2eb5aa02ad6d3dae9155154622e14e6cfcbe1cf78"
Jul 17 22:05:18.577885 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.577866    2563 scope.go:117] "RemoveContainer" containerID="ebe33c30714623993dac85ffb44ca87576d01f7e1563f402a1e06de7b137ba88"
Jul 17 22:05:18.578098 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:18.578077    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2_odh-ai-gateway-infra(bac73474-55e3-4852-a306-86c9e1d1e13e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e"
Jul 17 22:05:18.589261 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.589235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"]
Jul 17 22:05:18.720241 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.720215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:05:18.796536 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.796506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08419fb-c3f3-40a4-8edc-893dd5cab7c8-maas-api-service-ca\") pod \"f08419fb-c3f3-40a4-8edc-893dd5cab7c8\" (UID: \"f08419fb-c3f3-40a4-8edc-893dd5cab7c8\") "
Jul 17 22:05:18.796842 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.796820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08419fb-c3f3-40a4-8edc-893dd5cab7c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f08419fb-c3f3-40a4-8edc-893dd5cab7c8" (UID: "f08419fb-c3f3-40a4-8edc-893dd5cab7c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:18.897260 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:18.897227    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08419fb-c3f3-40a4-8edc-893dd5cab7c8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:05:19.583673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.583644    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"
Jul 17 22:05:19.583673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.583671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7" event={"ID":"f08419fb-c3f3-40a4-8edc-893dd5cab7c8","Type":"ContainerDied","Data":"44d57836a5e37e967b5dae95e0554bfaa6d93249a0985be6992f3125b95d73b9"}
Jul 17 22:05:19.584175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.583736    2563 scope.go:117] "RemoveContainer" containerID="52e40be30e9395aff9f7f08ac2a4a4d3e8cb38e22006c89f40c96e61c32a667b"
Jul 17 22:05:19.585726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.585696    2563 generic.go:356] "Generic (PLEG): container finished" podID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerID="c8034edc3a870aea4ab76bb222181761e8a8ba750d317df7fb36b3fd79e863a6" exitCode=6
Jul 17 22:05:19.585856 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.585749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" event={"ID":"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3","Type":"ContainerDied","Data":"c8034edc3a870aea4ab76bb222181761e8a8ba750d317df7fb36b3fd79e863a6"}
Jul 17 22:05:19.585944 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.585927    2563 scope.go:117] "RemoveContainer" containerID="c8034edc3a870aea4ab76bb222181761e8a8ba750d317df7fb36b3fd79e863a6"
Jul 17 22:05:19.586149 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:19.586131    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd_odh-ai-gateway-infra(c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3"
Jul 17 22:05:19.593324 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.593307    2563 scope.go:117] "RemoveContainer" containerID="69cd65f3004ac0a494c742a2eb5aa02ad6d3dae9155154622e14e6cfcbe1cf78"
Jul 17 22:05:19.616036 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.616014    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"]
Jul 17 22:05:19.619487 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:19.619468    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-7rnw7"]
Jul 17 22:05:20.593050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:20.593022    2563 scope.go:117] "RemoveContainer" containerID="c8034edc3a870aea4ab76bb222181761e8a8ba750d317df7fb36b3fd79e863a6"
Jul 17 22:05:20.593453 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:20.593219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd_odh-ai-gateway-infra(c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3"
Jul 17 22:05:20.601638 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:20.601613    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" path="/var/lib/kubelet/pods/f08419fb-c3f3-40a4-8edc-893dd5cab7c8/volumes"
Jul 17 22:05:29.596677 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:29.596644    2563 scope.go:117] "RemoveContainer" containerID="ebe33c30714623993dac85ffb44ca87576d01f7e1563f402a1e06de7b137ba88"
Jul 17 22:05:30.640369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:30.640324    2563 generic.go:356] "Generic (PLEG): container finished" podID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerID="a96970cc1e6fdae4e0c528bc84ddfcf492970d45f921f900ce43e211ceca6f30" exitCode=6
Jul 17 22:05:30.640822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:30.640396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" event={"ID":"bac73474-55e3-4852-a306-86c9e1d1e13e","Type":"ContainerDied","Data":"a96970cc1e6fdae4e0c528bc84ddfcf492970d45f921f900ce43e211ceca6f30"}
Jul 17 22:05:30.640822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:30.640453    2563 scope.go:117] "RemoveContainer" containerID="ebe33c30714623993dac85ffb44ca87576d01f7e1563f402a1e06de7b137ba88"
Jul 17 22:05:30.640822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:30.640810    2563 scope.go:117] "RemoveContainer" containerID="a96970cc1e6fdae4e0c528bc84ddfcf492970d45f921f900ce43e211ceca6f30"
Jul 17 22:05:30.641057 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:30.641037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2_odh-ai-gateway-infra(bac73474-55e3-4852-a306-86c9e1d1e13e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e"
Jul 17 22:05:31.596707 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:31.596677    2563 scope.go:117] "RemoveContainer" containerID="9809ff792ad9726d4360be8a29e56097ba03c31ac49e7e49b79aab9123bd83eb"
Jul 17 22:05:31.665615 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:31.665567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"]
Jul 17 22:05:31.830415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:31.830390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:31.897903 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:31.897873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac73474-55e3-4852-a306-86c9e1d1e13e-maas-api-service-ca\") pod \"bac73474-55e3-4852-a306-86c9e1d1e13e\" (UID: \"bac73474-55e3-4852-a306-86c9e1d1e13e\") "
Jul 17 22:05:31.898203 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:31.898181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac73474-55e3-4852-a306-86c9e1d1e13e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bac73474-55e3-4852-a306-86c9e1d1e13e" (UID: "bac73474-55e3-4852-a306-86c9e1d1e13e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:31.998470 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:31.998438    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac73474-55e3-4852-a306-86c9e1d1e13e-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:05:32.653897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.653867    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"
Jul 17 22:05:32.653897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.653882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2" event={"ID":"bac73474-55e3-4852-a306-86c9e1d1e13e","Type":"ContainerDied","Data":"d31bd4d1b87e06feaa2a608a6d5c8e29d12b0b45f858515234b92cccec1a53a4"}
Jul 17 22:05:32.654166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.653946    2563 scope.go:117] "RemoveContainer" containerID="a96970cc1e6fdae4e0c528bc84ddfcf492970d45f921f900ce43e211ceca6f30"
Jul 17 22:05:32.655923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.655892    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerID="3602d27ffafb9082d11d337c6c1003c84c372deb85582174bf89eba71df3ef82" exitCode=7
Jul 17 22:05:32.656093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.655967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" event={"ID":"7f65809a-152f-453b-897c-0fc4d6b06e9f","Type":"ContainerDied","Data":"3602d27ffafb9082d11d337c6c1003c84c372deb85582174bf89eba71df3ef82"}
Jul 17 22:05:32.656357 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.656298    2563 scope.go:117] "RemoveContainer" containerID="3602d27ffafb9082d11d337c6c1003c84c372deb85582174bf89eba71df3ef82"
Jul 17 22:05:32.656567 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:32.656541    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-x255s_odh-ai-gateway-infra(7f65809a-152f-453b-897c-0fc4d6b06e9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f"
Jul 17 22:05:32.663159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.663138    2563 scope.go:117] "RemoveContainer" containerID="9809ff792ad9726d4360be8a29e56097ba03c31ac49e7e49b79aab9123bd83eb"
Jul 17 22:05:32.687461 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.687435    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"]
Jul 17 22:05:32.689447 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:32.689425    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-w59c2"]
Jul 17 22:05:33.596985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:33.596958    2563 scope.go:117] "RemoveContainer" containerID="c8034edc3a870aea4ab76bb222181761e8a8ba750d317df7fb36b3fd79e863a6"
Jul 17 22:05:33.681782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:33.681743    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"]
Jul 17 22:05:33.832521 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:33.832492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:33.915311 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:33.915271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f65809a-152f-453b-897c-0fc4d6b06e9f-maas-api-service-ca\") pod \"7f65809a-152f-453b-897c-0fc4d6b06e9f\" (UID: \"7f65809a-152f-453b-897c-0fc4d6b06e9f\") "
Jul 17 22:05:33.915663 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:33.915639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f65809a-152f-453b-897c-0fc4d6b06e9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f65809a-152f-453b-897c-0fc4d6b06e9f" (UID: "7f65809a-152f-453b-897c-0fc4d6b06e9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:34.016146 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.016108    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f65809a-152f-453b-897c-0fc4d6b06e9f-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:05:34.603541 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.603505    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" path="/var/lib/kubelet/pods/bac73474-55e3-4852-a306-86c9e1d1e13e/volumes"
Jul 17 22:05:34.670864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.670826    2563 generic.go:356] "Generic (PLEG): container finished" podID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerID="d46b17b213cb3c2845d7e2d91a598865557ba3abc31ac8aeba86aa052631526c" exitCode=6
Jul 17 22:05:34.671052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.670896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" event={"ID":"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3","Type":"ContainerDied","Data":"d46b17b213cb3c2845d7e2d91a598865557ba3abc31ac8aeba86aa052631526c"}
Jul 17 22:05:34.671052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.670959    2563 scope.go:117] "RemoveContainer" containerID="c8034edc3a870aea4ab76bb222181761e8a8ba750d317df7fb36b3fd79e863a6"
Jul 17 22:05:34.671280 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.671258    2563 scope.go:117] "RemoveContainer" containerID="d46b17b213cb3c2845d7e2d91a598865557ba3abc31ac8aeba86aa052631526c"
Jul 17 22:05:34.671551 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:34.671517    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd_odh-ai-gateway-infra(c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3"
Jul 17 22:05:34.672422 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.672403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s" event={"ID":"7f65809a-152f-453b-897c-0fc4d6b06e9f","Type":"ContainerDied","Data":"c14582c4cd8aff009283c7cc3ff267520c7f0406d74ab7ede849f34bcf1d7401"}
Jul 17 22:05:34.672422 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.672419    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"
Jul 17 22:05:34.683176 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.683019    2563 scope.go:117] "RemoveContainer" containerID="3602d27ffafb9082d11d337c6c1003c84c372deb85582174bf89eba71df3ef82"
Jul 17 22:05:34.698253 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.698226    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"]
Jul 17 22:05:34.704686 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:34.704664    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-x255s"]
Jul 17 22:05:35.695516 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:35.695484    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"]
Jul 17 22:05:35.838174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:35.838148    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:35.932548 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:35.932517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3-maas-api-service-ca\") pod \"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3\" (UID: \"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3\") "
Jul 17 22:05:35.932918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:35.932893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" (UID: "c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:36.033837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.033746    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:05:36.603122 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.603090    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" path="/var/lib/kubelet/pods/7f65809a-152f-453b-897c-0fc4d6b06e9f/volumes"
Jul 17 22:05:36.685987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.685958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"
Jul 17 22:05:36.686191 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.685981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd" event={"ID":"c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3","Type":"ContainerDied","Data":"b73eae9ec274ef31e66d18e20e9c43c9b9350ad60d993e4ec3e71dc15f23c40b"}
Jul 17 22:05:36.686191 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.686052    2563 scope.go:117] "RemoveContainer" containerID="d46b17b213cb3c2845d7e2d91a598865557ba3abc31ac8aeba86aa052631526c"
Jul 17 22:05:36.700406 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.700371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"]
Jul 17 22:05:36.705692 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:36.705668    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-zzwgd"]
Jul 17 22:05:38.602702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:38.602667    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" path="/var/lib/kubelet/pods/c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3/volumes"
Jul 17 22:05:49.702458 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.702414    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"]
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.702960    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.702982    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703036    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703048    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703057    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703078    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703088    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703097    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703107    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703115    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703130    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703138    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703146    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703153    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703183    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703191    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703284    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703307    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703317    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703326    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703334    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703343    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703352    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f65809a-152f-453b-897c-0fc4d6b06e9f" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703360    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:05:49.703869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.703367    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83308f8-00cd-46e6-a3bc-d9c13b6ac4d3" containerName="revoke-keys"
Jul 17 22:05:49.708094 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.708072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:05:49.711181 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.711159    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:05:49.714394 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.714286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"]
Jul 17 22:05:49.754312 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.754280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e5624d-1e37-40fb-b901-cc7f12f54855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw\" (UID: \"44e5624d-1e37-40fb-b901-cc7f12f54855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:05:49.855304 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.855271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e5624d-1e37-40fb-b901-cc7f12f54855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw\" (UID: \"44e5624d-1e37-40fb-b901-cc7f12f54855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:05:49.855962 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:49.855943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e5624d-1e37-40fb-b901-cc7f12f54855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw\" (UID: \"44e5624d-1e37-40fb-b901-cc7f12f54855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:05:50.020985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:50.020884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:05:50.161061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:50.161034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"]
Jul 17 22:05:50.163310 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:05:50.163282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e5624d_1e37_40fb_b901_cc7f12f54855.slice/crio-3214516e45bdf5e74a85007eb54f5b4c3efa6c010798256bf9e94f727023f0a3 WatchSource:0}: Error finding container 3214516e45bdf5e74a85007eb54f5b4c3efa6c010798256bf9e94f727023f0a3: Status 404 returned error can't find the container with id 3214516e45bdf5e74a85007eb54f5b4c3efa6c010798256bf9e94f727023f0a3
Jul 17 22:05:50.758102 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:50.758065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" event={"ID":"44e5624d-1e37-40fb-b901-cc7f12f54855","Type":"ContainerStarted","Data":"17d8cacd8048ff84ea80280ec968902201baf2a25f501c529479aa24b7a0d9e0"}
Jul 17 22:05:50.758562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:50.758131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" event={"ID":"44e5624d-1e37-40fb-b901-cc7f12f54855","Type":"ContainerStarted","Data":"3214516e45bdf5e74a85007eb54f5b4c3efa6c010798256bf9e94f727023f0a3"}
Jul 17 22:05:50.773587 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:50.773533    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" podStartSLOduration=1.773517544 podStartE2EDuration="1.773517544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:50.77254944 +0000 UTC m=+2314.684789730" watchObservedRunningTime="2026-07-17 22:05:50.773517544 +0000 UTC m=+2314.685757833"
Jul 17 22:05:51.763908 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:51.763871    2563 generic.go:356] "Generic (PLEG): container finished" podID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerID="17d8cacd8048ff84ea80280ec968902201baf2a25f501c529479aa24b7a0d9e0" exitCode=6
Jul 17 22:05:51.764320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:51.763929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" event={"ID":"44e5624d-1e37-40fb-b901-cc7f12f54855","Type":"ContainerDied","Data":"17d8cacd8048ff84ea80280ec968902201baf2a25f501c529479aa24b7a0d9e0"}
Jul 17 22:05:51.764320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:51.764128    2563 scope.go:117] "RemoveContainer" containerID="17d8cacd8048ff84ea80280ec968902201baf2a25f501c529479aa24b7a0d9e0"
Jul 17 22:05:52.771023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:52.770990    2563 generic.go:356] "Generic (PLEG): container finished" podID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerID="1bc7cf90f2dad1275bc451bba7343139e60e8c7a3e97ba3f8875f83793595832" exitCode=6
Jul 17 22:05:52.771431 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:52.771069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" event={"ID":"44e5624d-1e37-40fb-b901-cc7f12f54855","Type":"ContainerDied","Data":"1bc7cf90f2dad1275bc451bba7343139e60e8c7a3e97ba3f8875f83793595832"}
Jul 17 22:05:52.771431 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:52.771123    2563 scope.go:117] "RemoveContainer" containerID="17d8cacd8048ff84ea80280ec968902201baf2a25f501c529479aa24b7a0d9e0"
Jul 17 22:05:52.771431 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:52.771346    2563 scope.go:117] "RemoveContainer" containerID="1bc7cf90f2dad1275bc451bba7343139e60e8c7a3e97ba3f8875f83793595832"
Jul 17 22:05:52.771575 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:52.771555    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw_odh-ai-gateway-infra(44e5624d-1e37-40fb-b901-cc7f12f54855)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855"
Jul 17 22:05:53.777832 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:05:53.777803    2563 scope.go:117] "RemoveContainer" containerID="1bc7cf90f2dad1275bc451bba7343139e60e8c7a3e97ba3f8875f83793595832"
Jul 17 22:05:53.778212 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:05:53.777987    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw_odh-ai-gateway-infra(44e5624d-1e37-40fb-b901-cc7f12f54855)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855"
Jul 17 22:06:05.372537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.372501    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"]
Jul 17 22:06:05.372957 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.372942    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:06:05.373002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.372959    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:06:05.373057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.373048    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac73474-55e3-4852-a306-86c9e1d1e13e" containerName="revoke-keys"
Jul 17 22:06:05.373090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.373062    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08419fb-c3f3-40a4-8edc-893dd5cab7c8" containerName="revoke-keys"
Jul 17 22:06:05.377469 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.377451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:05.382668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.382644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"]
Jul 17 22:06:05.500866 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.500825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/229b928e-d63a-47f3-8e2a-ede2d11845bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x\" (UID: \"229b928e-d63a-47f3-8e2a-ede2d11845bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:05.597451 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.597427    2563 scope.go:117] "RemoveContainer" containerID="1bc7cf90f2dad1275bc451bba7343139e60e8c7a3e97ba3f8875f83793595832"
Jul 17 22:06:05.601317 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.601296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/229b928e-d63a-47f3-8e2a-ede2d11845bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x\" (UID: \"229b928e-d63a-47f3-8e2a-ede2d11845bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:05.601923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.601906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/229b928e-d63a-47f3-8e2a-ede2d11845bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x\" (UID: \"229b928e-d63a-47f3-8e2a-ede2d11845bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:05.689590 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:05.689557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:06.037354 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.037330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"]
Jul 17 22:06:06.039009 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:06:06.038978    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229b928e_d63a_47f3_8e2a_ede2d11845bf.slice/crio-10b601055c6fd2d0e4e9b22cb8ce3e63b906473f10b64f27b43e8b76402ab9b4 WatchSource:0}: Error finding container 10b601055c6fd2d0e4e9b22cb8ce3e63b906473f10b64f27b43e8b76402ab9b4: Status 404 returned error can't find the container with id 10b601055c6fd2d0e4e9b22cb8ce3e63b906473f10b64f27b43e8b76402ab9b4
Jul 17 22:06:06.840705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.840668    2563 generic.go:356] "Generic (PLEG): container finished" podID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerID="df663e4b870faa9bf5680cde784245d84aad827baebfa2077b56074cf75cd03e" exitCode=6
Jul 17 22:06:06.841175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.840752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" event={"ID":"229b928e-d63a-47f3-8e2a-ede2d11845bf","Type":"ContainerDied","Data":"df663e4b870faa9bf5680cde784245d84aad827baebfa2077b56074cf75cd03e"}
Jul 17 22:06:06.841175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.840804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" event={"ID":"229b928e-d63a-47f3-8e2a-ede2d11845bf","Type":"ContainerStarted","Data":"10b601055c6fd2d0e4e9b22cb8ce3e63b906473f10b64f27b43e8b76402ab9b4"}
Jul 17 22:06:06.841175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.840958    2563 scope.go:117] "RemoveContainer" containerID="df663e4b870faa9bf5680cde784245d84aad827baebfa2077b56074cf75cd03e"
Jul 17 22:06:06.842665 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.842645    2563 generic.go:356] "Generic (PLEG): container finished" podID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerID="0cd60ca81d3516c2892eef55cc2673a0eab7d03df163de11b728d84be2c41ec6" exitCode=6
Jul 17 22:06:06.842765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.842724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" event={"ID":"44e5624d-1e37-40fb-b901-cc7f12f54855","Type":"ContainerDied","Data":"0cd60ca81d3516c2892eef55cc2673a0eab7d03df163de11b728d84be2c41ec6"}
Jul 17 22:06:06.842804 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.842770    2563 scope.go:117] "RemoveContainer" containerID="1bc7cf90f2dad1275bc451bba7343139e60e8c7a3e97ba3f8875f83793595832"
Jul 17 22:06:06.843057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:06.843033    2563 scope.go:117] "RemoveContainer" containerID="0cd60ca81d3516c2892eef55cc2673a0eab7d03df163de11b728d84be2c41ec6"
Jul 17 22:06:06.843247 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:06.843223    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw_odh-ai-gateway-infra(44e5624d-1e37-40fb-b901-cc7f12f54855)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855"
Jul 17 22:06:07.554050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.554021    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"]
Jul 17 22:06:07.557534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.557516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:07.576878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.576851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"]
Jul 17 22:06:07.718712 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.718671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42711b5-2a4e-49e9-b4f0-0985b46bd837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh\" (UID: \"b42711b5-2a4e-49e9-b4f0-0985b46bd837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:07.820159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.820036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42711b5-2a4e-49e9-b4f0-0985b46bd837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh\" (UID: \"b42711b5-2a4e-49e9-b4f0-0985b46bd837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:07.820717 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.820698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42711b5-2a4e-49e9-b4f0-0985b46bd837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh\" (UID: \"b42711b5-2a4e-49e9-b4f0-0985b46bd837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:07.849415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.849389    2563 generic.go:356] "Generic (PLEG): container finished" podID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerID="ae38f9310d89c40562c515a9c3b760aa37da488971931ed09656539d48204c78" exitCode=6
Jul 17 22:06:07.849862 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.849476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" event={"ID":"229b928e-d63a-47f3-8e2a-ede2d11845bf","Type":"ContainerDied","Data":"ae38f9310d89c40562c515a9c3b760aa37da488971931ed09656539d48204c78"}
Jul 17 22:06:07.849862 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.849533    2563 scope.go:117] "RemoveContainer" containerID="df663e4b870faa9bf5680cde784245d84aad827baebfa2077b56074cf75cd03e"
Jul 17 22:06:07.849862 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.849723    2563 scope.go:117] "RemoveContainer" containerID="ae38f9310d89c40562c515a9c3b760aa37da488971931ed09656539d48204c78"
Jul 17 22:06:07.849988 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:07.849950    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x_odh-ai-gateway-infra(229b928e-d63a-47f3-8e2a-ede2d11845bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf"
Jul 17 22:06:07.879260 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.879237    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"]
Jul 17 22:06:07.920488 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:07.920454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:08.029261 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.029231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:06:08.069532 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.069497    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"]
Jul 17 22:06:08.072417 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:06:08.072390    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42711b5_2a4e_49e9_b4f0_0985b46bd837.slice/crio-640f2d6d941282b47bfad34b4ce7c5883a0fc0dd41c82d2fa89e1653f56dc551 WatchSource:0}: Error finding container 640f2d6d941282b47bfad34b4ce7c5883a0fc0dd41c82d2fa89e1653f56dc551: Status 404 returned error can't find the container with id 640f2d6d941282b47bfad34b4ce7c5883a0fc0dd41c82d2fa89e1653f56dc551
Jul 17 22:06:08.122246 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.122212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e5624d-1e37-40fb-b901-cc7f12f54855-maas-api-service-ca\") pod \"44e5624d-1e37-40fb-b901-cc7f12f54855\" (UID: \"44e5624d-1e37-40fb-b901-cc7f12f54855\") "
Jul 17 22:06:08.122556 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.122535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e5624d-1e37-40fb-b901-cc7f12f54855-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44e5624d-1e37-40fb-b901-cc7f12f54855" (UID: "44e5624d-1e37-40fb-b901-cc7f12f54855"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:08.223681 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.223646    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e5624d-1e37-40fb-b901-cc7f12f54855-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:06:08.857040 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.857010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"
Jul 17 22:06:08.857040 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.857028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw" event={"ID":"44e5624d-1e37-40fb-b901-cc7f12f54855","Type":"ContainerDied","Data":"3214516e45bdf5e74a85007eb54f5b4c3efa6c010798256bf9e94f727023f0a3"}
Jul 17 22:06:08.857562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.857094    2563 scope.go:117] "RemoveContainer" containerID="0cd60ca81d3516c2892eef55cc2673a0eab7d03df163de11b728d84be2c41ec6"
Jul 17 22:06:08.858701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.858674    2563 generic.go:356] "Generic (PLEG): container finished" podID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerID="315e9026436ccf66525e2f58f20e2ca4a8d14c5f90109c28bb33d388a04e937f" exitCode=6
Jul 17 22:06:08.858837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.858740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" event={"ID":"b42711b5-2a4e-49e9-b4f0-0985b46bd837","Type":"ContainerDied","Data":"315e9026436ccf66525e2f58f20e2ca4a8d14c5f90109c28bb33d388a04e937f"}
Jul 17 22:06:08.858837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.858795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" event={"ID":"b42711b5-2a4e-49e9-b4f0-0985b46bd837","Type":"ContainerStarted","Data":"640f2d6d941282b47bfad34b4ce7c5883a0fc0dd41c82d2fa89e1653f56dc551"}
Jul 17 22:06:08.858976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.858911    2563 scope.go:117] "RemoveContainer" containerID="315e9026436ccf66525e2f58f20e2ca4a8d14c5f90109c28bb33d388a04e937f"
Jul 17 22:06:08.860777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.860752    2563 scope.go:117] "RemoveContainer" containerID="ae38f9310d89c40562c515a9c3b760aa37da488971931ed09656539d48204c78"
Jul 17 22:06:08.860954 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:08.860933    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x_odh-ai-gateway-infra(229b928e-d63a-47f3-8e2a-ede2d11845bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf"
Jul 17 22:06:08.872626 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.872586    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"]
Jul 17 22:06:08.878523 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:08.878503    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9hqrw"]
Jul 17 22:06:09.867585 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:09.867499    2563 generic.go:356] "Generic (PLEG): container finished" podID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerID="a407fa2d472599a8d19009ffda9014627e6ddf181c77eec6cb2871f84fe04fd1" exitCode=6
Jul 17 22:06:09.867986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:09.867582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" event={"ID":"b42711b5-2a4e-49e9-b4f0-0985b46bd837","Type":"ContainerDied","Data":"a407fa2d472599a8d19009ffda9014627e6ddf181c77eec6cb2871f84fe04fd1"}
Jul 17 22:06:09.867986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:09.867695    2563 scope.go:117] "RemoveContainer" containerID="315e9026436ccf66525e2f58f20e2ca4a8d14c5f90109c28bb33d388a04e937f"
Jul 17 22:06:09.867986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:09.867788    2563 scope.go:117] "RemoveContainer" containerID="a407fa2d472599a8d19009ffda9014627e6ddf181c77eec6cb2871f84fe04fd1"
Jul 17 22:06:09.868106 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:09.867983    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh_odh-ai-gateway-infra(b42711b5-2a4e-49e9-b4f0-0985b46bd837)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837"
Jul 17 22:06:10.602701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:10.602672    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" path="/var/lib/kubelet/pods/44e5624d-1e37-40fb-b901-cc7f12f54855/volumes"
Jul 17 22:06:10.874214 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:10.874141    2563 scope.go:117] "RemoveContainer" containerID="a407fa2d472599a8d19009ffda9014627e6ddf181c77eec6cb2871f84fe04fd1"
Jul 17 22:06:10.874557 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:10.874342    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh_odh-ai-gateway-infra(b42711b5-2a4e-49e9-b4f0-0985b46bd837)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837"
Jul 17 22:06:12.890235 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890194    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"]
Jul 17 22:06:12.890764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890744    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.890843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890767    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.890843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890785    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.890843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.890843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890817    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.890843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890827    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.891112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890932    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.891112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.890945    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:12.895359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.895339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:12.901697 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:12.901667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"]
Jul 17 22:06:13.067102 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.067067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e21376b-01d2-4566-89d3-27a749895c07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp\" (UID: \"0e21376b-01d2-4566-89d3-27a749895c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:13.167912 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.167876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e21376b-01d2-4566-89d3-27a749895c07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp\" (UID: \"0e21376b-01d2-4566-89d3-27a749895c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:13.168482 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.168464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e21376b-01d2-4566-89d3-27a749895c07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp\" (UID: \"0e21376b-01d2-4566-89d3-27a749895c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:13.207421 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.207387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:13.345152 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.345123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"]
Jul 17 22:06:13.890223 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.890128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerStarted","Data":"3998d8226e4e661392303b48e2fa1ed30a8a375a6d847aed31debb521cad5e69"}
Jul 17 22:06:13.890223 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.890210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerStarted","Data":"f8dc53032e4548cecf55f5f7c38ea39e6fb165a47792d4f69e7de1b99844634d"}
Jul 17 22:06:13.907835 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:13.907781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" podStartSLOduration=1.907765282 podStartE2EDuration="1.907765282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:13.905657761 +0000 UTC m=+2337.817898051" watchObservedRunningTime="2026-07-17 22:06:13.907765282 +0000 UTC m=+2337.820005918"
Jul 17 22:06:14.895843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:14.895808    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e21376b-01d2-4566-89d3-27a749895c07" containerID="3998d8226e4e661392303b48e2fa1ed30a8a375a6d847aed31debb521cad5e69" exitCode=7
Jul 17 22:06:14.896228 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:14.895890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerDied","Data":"3998d8226e4e661392303b48e2fa1ed30a8a375a6d847aed31debb521cad5e69"}
Jul 17 22:06:14.896228 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:14.896059    2563 scope.go:117] "RemoveContainer" containerID="3998d8226e4e661392303b48e2fa1ed30a8a375a6d847aed31debb521cad5e69"
Jul 17 22:06:15.905412 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:15.905328    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e21376b-01d2-4566-89d3-27a749895c07" containerID="d3dc86266e35f48a79e747a5aa67a157b3fafda6896d3883e028eec59f3689b1" exitCode=7
Jul 17 22:06:15.905892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:15.905407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerDied","Data":"d3dc86266e35f48a79e747a5aa67a157b3fafda6896d3883e028eec59f3689b1"}
Jul 17 22:06:15.905892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:15.905464    2563 scope.go:117] "RemoveContainer" containerID="3998d8226e4e661392303b48e2fa1ed30a8a375a6d847aed31debb521cad5e69"
Jul 17 22:06:15.905892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:15.905725    2563 scope.go:117] "RemoveContainer" containerID="d3dc86266e35f48a79e747a5aa67a157b3fafda6896d3883e028eec59f3689b1"
Jul 17 22:06:15.906018 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:15.905935    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp_odh-ai-gateway-infra(0e21376b-01d2-4566-89d3-27a749895c07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" podUID="0e21376b-01d2-4566-89d3-27a749895c07"
Jul 17 22:06:16.911922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:16.911849    2563 scope.go:117] "RemoveContainer" containerID="d3dc86266e35f48a79e747a5aa67a157b3fafda6896d3883e028eec59f3689b1"
Jul 17 22:06:16.912193 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:16.912071    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp_odh-ai-gateway-infra(0e21376b-01d2-4566-89d3-27a749895c07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" podUID="0e21376b-01d2-4566-89d3-27a749895c07"
Jul 17 22:06:20.596890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:20.596855    2563 scope.go:117] "RemoveContainer" containerID="ae38f9310d89c40562c515a9c3b760aa37da488971931ed09656539d48204c78"
Jul 17 22:06:20.930721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:20.930690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" event={"ID":"229b928e-d63a-47f3-8e2a-ede2d11845bf","Type":"ContainerStarted","Data":"b7e2fcf9db5fdff03f59d5b1654ec07f6a0a6e39bb260282d674a6f4bec09b0b"}
Jul 17 22:06:20.946805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:20.946759    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" podStartSLOduration=15.946744304 podStartE2EDuration="15.946744304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:20.945781992 +0000 UTC m=+2344.858022282" watchObservedRunningTime="2026-07-17 22:06:20.946744304 +0000 UTC m=+2344.858984592"
Jul 17 22:06:21.937767 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:21.937733    2563 generic.go:356] "Generic (PLEG): container finished" podID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerID="b7e2fcf9db5fdff03f59d5b1654ec07f6a0a6e39bb260282d674a6f4bec09b0b" exitCode=6
Jul 17 22:06:21.938203 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:21.937806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" event={"ID":"229b928e-d63a-47f3-8e2a-ede2d11845bf","Type":"ContainerDied","Data":"b7e2fcf9db5fdff03f59d5b1654ec07f6a0a6e39bb260282d674a6f4bec09b0b"}
Jul 17 22:06:21.938203 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:21.937864    2563 scope.go:117] "RemoveContainer" containerID="ae38f9310d89c40562c515a9c3b760aa37da488971931ed09656539d48204c78"
Jul 17 22:06:21.938203 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:21.938118    2563 scope.go:117] "RemoveContainer" containerID="b7e2fcf9db5fdff03f59d5b1654ec07f6a0a6e39bb260282d674a6f4bec09b0b"
Jul 17 22:06:21.938356 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:21.938334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x_odh-ai-gateway-infra(229b928e-d63a-47f3-8e2a-ede2d11845bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf"
Jul 17 22:06:21.963008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:21.962985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"]
Jul 17 22:06:22.596872 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:22.596842    2563 scope.go:117] "RemoveContainer" containerID="a407fa2d472599a8d19009ffda9014627e6ddf181c77eec6cb2871f84fe04fd1"
Jul 17 22:06:22.943947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:22.943908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" event={"ID":"b42711b5-2a4e-49e9-b4f0-0985b46bd837","Type":"ContainerStarted","Data":"06c4a2a76309195dacbbb477155071c6c356218db449594e18edb30b4cb2c32f"}
Jul 17 22:06:22.959919 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:22.959851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" podStartSLOduration=15.959829065 podStartE2EDuration="15.959829065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:22.959382595 +0000 UTC m=+2346.871622884" watchObservedRunningTime="2026-07-17 22:06:22.959829065 +0000 UTC m=+2346.872069355"
Jul 17 22:06:23.095092 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.095069    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:23.154063 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.154031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/229b928e-d63a-47f3-8e2a-ede2d11845bf-maas-api-service-ca\") pod \"229b928e-d63a-47f3-8e2a-ede2d11845bf\" (UID: \"229b928e-d63a-47f3-8e2a-ede2d11845bf\") "
Jul 17 22:06:23.154410 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.154388    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229b928e-d63a-47f3-8e2a-ede2d11845bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "229b928e-d63a-47f3-8e2a-ede2d11845bf" (UID: "229b928e-d63a-47f3-8e2a-ede2d11845bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:23.255132 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.255110    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/229b928e-d63a-47f3-8e2a-ede2d11845bf-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:06:23.952028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.951999    2563 generic.go:356] "Generic (PLEG): container finished" podID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerID="06c4a2a76309195dacbbb477155071c6c356218db449594e18edb30b4cb2c32f" exitCode=6
Jul 17 22:06:23.952497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.952069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" event={"ID":"b42711b5-2a4e-49e9-b4f0-0985b46bd837","Type":"ContainerDied","Data":"06c4a2a76309195dacbbb477155071c6c356218db449594e18edb30b4cb2c32f"}
Jul 17 22:06:23.952497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.952130    2563 scope.go:117] "RemoveContainer" containerID="a407fa2d472599a8d19009ffda9014627e6ddf181c77eec6cb2871f84fe04fd1"
Jul 17 22:06:23.952497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.952434    2563 scope.go:117] "RemoveContainer" containerID="06c4a2a76309195dacbbb477155071c6c356218db449594e18edb30b4cb2c32f"
Jul 17 22:06:23.952731 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:23.952711    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh_odh-ai-gateway-infra(b42711b5-2a4e-49e9-b4f0-0985b46bd837)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837"
Jul 17 22:06:23.953500 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.953485    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"
Jul 17 22:06:23.953587 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.953529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x" event={"ID":"229b928e-d63a-47f3-8e2a-ede2d11845bf","Type":"ContainerDied","Data":"10b601055c6fd2d0e4e9b22cb8ce3e63b906473f10b64f27b43e8b76402ab9b4"}
Jul 17 22:06:23.964459 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.964438    2563 scope.go:117] "RemoveContainer" containerID="b7e2fcf9db5fdff03f59d5b1654ec07f6a0a6e39bb260282d674a6f4bec09b0b"
Jul 17 22:06:23.974107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.971667    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"]
Jul 17 22:06:23.992772 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.992738    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"]
Jul 17 22:06:23.999813 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:23.999787    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qkg6x"]
Jul 17 22:06:24.602231 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:24.602193    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" path="/var/lib/kubelet/pods/229b928e-d63a-47f3-8e2a-ede2d11845bf/volumes"
Jul 17 22:06:25.085648 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.085622    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:25.172384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.172350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42711b5-2a4e-49e9-b4f0-0985b46bd837-maas-api-service-ca\") pod \"b42711b5-2a4e-49e9-b4f0-0985b46bd837\" (UID: \"b42711b5-2a4e-49e9-b4f0-0985b46bd837\") "
Jul 17 22:06:25.172799 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.172772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42711b5-2a4e-49e9-b4f0-0985b46bd837-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b42711b5-2a4e-49e9-b4f0-0985b46bd837" (UID: "b42711b5-2a4e-49e9-b4f0-0985b46bd837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:25.273202 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.273115    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42711b5-2a4e-49e9-b4f0-0985b46bd837-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:06:25.967586 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.967562    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"
Jul 17 22:06:25.967586 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.967567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh" event={"ID":"b42711b5-2a4e-49e9-b4f0-0985b46bd837","Type":"ContainerDied","Data":"640f2d6d941282b47bfad34b4ce7c5883a0fc0dd41c82d2fa89e1653f56dc551"}
Jul 17 22:06:25.967881 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.967644    2563 scope.go:117] "RemoveContainer" containerID="06c4a2a76309195dacbbb477155071c6c356218db449594e18edb30b4cb2c32f"
Jul 17 22:06:25.988434 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.988406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"]
Jul 17 22:06:25.994307 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:25.994282    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-h9zjh"]
Jul 17 22:06:26.602656 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:26.602622    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" path="/var/lib/kubelet/pods/b42711b5-2a4e-49e9-b4f0-0985b46bd837/volumes"
Jul 17 22:06:31.597058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:31.597027    2563 scope.go:117] "RemoveContainer" containerID="d3dc86266e35f48a79e747a5aa67a157b3fafda6896d3883e028eec59f3689b1"
Jul 17 22:06:31.999697 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:31.999665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerStarted","Data":"45f35f16caa3af7023534f2c57a5d16dbd0b7443732071406165be203f1cfd04"}
Jul 17 22:06:33.005897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:33.005861    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e21376b-01d2-4566-89d3-27a749895c07" containerID="45f35f16caa3af7023534f2c57a5d16dbd0b7443732071406165be203f1cfd04" exitCode=7
Jul 17 22:06:33.006294 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:33.005931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerDied","Data":"45f35f16caa3af7023534f2c57a5d16dbd0b7443732071406165be203f1cfd04"}
Jul 17 22:06:33.006294 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:33.005993    2563 scope.go:117] "RemoveContainer" containerID="d3dc86266e35f48a79e747a5aa67a157b3fafda6896d3883e028eec59f3689b1"
Jul 17 22:06:33.006381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:33.006340    2563 scope.go:117] "RemoveContainer" containerID="45f35f16caa3af7023534f2c57a5d16dbd0b7443732071406165be203f1cfd04"
Jul 17 22:06:33.006566 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:33.006549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp_odh-ai-gateway-infra(0e21376b-01d2-4566-89d3-27a749895c07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" podUID="0e21376b-01d2-4566-89d3-27a749895c07"
Jul 17 22:06:33.021941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:33.021915    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"]
Jul 17 22:06:34.155448 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:34.155421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:34.256816 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:34.256771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e21376b-01d2-4566-89d3-27a749895c07-maas-api-service-ca\") pod \"0e21376b-01d2-4566-89d3-27a749895c07\" (UID: \"0e21376b-01d2-4566-89d3-27a749895c07\") "
Jul 17 22:06:34.257093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:34.257070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e21376b-01d2-4566-89d3-27a749895c07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e21376b-01d2-4566-89d3-27a749895c07" (UID: "0e21376b-01d2-4566-89d3-27a749895c07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:34.357740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:34.357668    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e21376b-01d2-4566-89d3-27a749895c07-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:06:35.019141 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:35.019100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp" event={"ID":"0e21376b-01d2-4566-89d3-27a749895c07","Type":"ContainerDied","Data":"f8dc53032e4548cecf55f5f7c38ea39e6fb165a47792d4f69e7de1b99844634d"}
Jul 17 22:06:35.019141 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:35.019140    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"
Jul 17 22:06:35.019410 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:35.019176    2563 scope.go:117] "RemoveContainer" containerID="45f35f16caa3af7023534f2c57a5d16dbd0b7443732071406165be203f1cfd04"
Jul 17 22:06:35.036026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:35.035994    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"]
Jul 17 22:06:35.041876 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:35.041837    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kp7vp"]
Jul 17 22:06:36.602672 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:36.602640    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e21376b-01d2-4566-89d3-27a749895c07" path="/var/lib/kubelet/pods/0e21376b-01d2-4566-89d3-27a749895c07/volumes"
Jul 17 22:06:39.741142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741108    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"]
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741455    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741466    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741493    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741501    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741516    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741521    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741535    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.741539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741546    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741562    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741572    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741648    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741657    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741663    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741668    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741673    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e5624d-1e37-40fb-b901-cc7f12f54855" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741678    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741684    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741690    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42711b5-2a4e-49e9-b4f0-0985b46bd837" containerName="revoke-keys"
Jul 17 22:06:39.742062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.741716    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e21376b-01d2-4566-89d3-27a749895c07" containerName="revoke-keys"
Jul 17 22:06:39.746231 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.746213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:39.749107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.749082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:06:39.751647 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.751620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"]
Jul 17 22:06:39.806546 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.806519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f4a191-f64a-4583-8590-c13857cfbc9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt\" (UID: \"83f4a191-f64a-4583-8590-c13857cfbc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:39.907450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.907411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f4a191-f64a-4583-8590-c13857cfbc9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt\" (UID: \"83f4a191-f64a-4583-8590-c13857cfbc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:39.908041 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:39.908015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f4a191-f64a-4583-8590-c13857cfbc9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt\" (UID: \"83f4a191-f64a-4583-8590-c13857cfbc9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:40.058776 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:40.058690    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:40.193120 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:40.193095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"]
Jul 17 22:06:40.195972 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:06:40.195942    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f4a191_f64a_4583_8590_c13857cfbc9d.slice/crio-e3cd87993ed47c66c606dddfc96d818274b96d34148b18b787e7d1fe34dac7d7 WatchSource:0}: Error finding container e3cd87993ed47c66c606dddfc96d818274b96d34148b18b787e7d1fe34dac7d7: Status 404 returned error can't find the container with id e3cd87993ed47c66c606dddfc96d818274b96d34148b18b787e7d1fe34dac7d7
Jul 17 22:06:41.051522 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:41.051480    2563 generic.go:356] "Generic (PLEG): container finished" podID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerID="b092a9d0c7b7629299ad840ea891b7b7d0902915bb493f7d15f5cd2d8d3e21cf" exitCode=6
Jul 17 22:06:41.051928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:41.051521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" event={"ID":"83f4a191-f64a-4583-8590-c13857cfbc9d","Type":"ContainerDied","Data":"b092a9d0c7b7629299ad840ea891b7b7d0902915bb493f7d15f5cd2d8d3e21cf"}
Jul 17 22:06:41.051928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:41.051584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" event={"ID":"83f4a191-f64a-4583-8590-c13857cfbc9d","Type":"ContainerStarted","Data":"e3cd87993ed47c66c606dddfc96d818274b96d34148b18b787e7d1fe34dac7d7"}
Jul 17 22:06:41.051928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:41.051813    2563 scope.go:117] "RemoveContainer" containerID="b092a9d0c7b7629299ad840ea891b7b7d0902915bb493f7d15f5cd2d8d3e21cf"
Jul 17 22:06:42.058663 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:42.058629    2563 generic.go:356] "Generic (PLEG): container finished" podID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerID="280f82df7d6a2d1ba8f0c643c79920e3e6cbb8a39f80f542aedb6e2f4c7856f5" exitCode=6
Jul 17 22:06:42.059122 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:42.058706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" event={"ID":"83f4a191-f64a-4583-8590-c13857cfbc9d","Type":"ContainerDied","Data":"280f82df7d6a2d1ba8f0c643c79920e3e6cbb8a39f80f542aedb6e2f4c7856f5"}
Jul 17 22:06:42.059122 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:42.058763    2563 scope.go:117] "RemoveContainer" containerID="b092a9d0c7b7629299ad840ea891b7b7d0902915bb493f7d15f5cd2d8d3e21cf"
Jul 17 22:06:42.059122 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:42.058914    2563 scope.go:117] "RemoveContainer" containerID="280f82df7d6a2d1ba8f0c643c79920e3e6cbb8a39f80f542aedb6e2f4c7856f5"
Jul 17 22:06:42.059244 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:42.059125    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt_odh-ai-gateway-infra(83f4a191-f64a-4583-8590-c13857cfbc9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d"
Jul 17 22:06:43.065016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:43.064971    2563 scope.go:117] "RemoveContainer" containerID="280f82df7d6a2d1ba8f0c643c79920e3e6cbb8a39f80f542aedb6e2f4c7856f5"
Jul 17 22:06:43.065494 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:43.065201    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt_odh-ai-gateway-infra(83f4a191-f64a-4583-8590-c13857cfbc9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d"
Jul 17 22:06:54.597572 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:54.597534    2563 scope.go:117] "RemoveContainer" containerID="280f82df7d6a2d1ba8f0c643c79920e3e6cbb8a39f80f542aedb6e2f4c7856f5"
Jul 17 22:06:55.122578 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.122540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" event={"ID":"83f4a191-f64a-4583-8590-c13857cfbc9d","Type":"ContainerStarted","Data":"31db861116aca2193eb18dc32da0bf5b8599f0a4f387d9fd2cd5ce2f65d4dfb4"}
Jul 17 22:06:55.138078 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.137919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" podStartSLOduration=16.13787946 podStartE2EDuration="16.13787946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:55.137818612 +0000 UTC m=+2379.050058912" watchObservedRunningTime="2026-07-17 22:06:55.13787946 +0000 UTC m=+2379.050119750"
Jul 17 22:06:55.489654 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.489621    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"]
Jul 17 22:06:55.490072 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.490060    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="229b928e-d63a-47f3-8e2a-ede2d11845bf" containerName="revoke-keys"
Jul 17 22:06:55.493053 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.493037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:06:55.500183 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.500155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"]
Jul 17 22:06:55.652961 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.652931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64013b03-317c-4575-82bc-52f333057f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr\" (UID: \"64013b03-317c-4575-82bc-52f333057f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:06:55.753664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.753554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64013b03-317c-4575-82bc-52f333057f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr\" (UID: \"64013b03-317c-4575-82bc-52f333057f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:06:55.754160 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.754139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64013b03-317c-4575-82bc-52f333057f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr\" (UID: \"64013b03-317c-4575-82bc-52f333057f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:06:55.804584 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.804554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:06:55.942566 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:55.942536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"]
Jul 17 22:06:55.944100 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:06:55.944075    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64013b03_317c_4575_82bc_52f333057f11.slice/crio-3a3b4e44997119135b104ebe5a22290525b7bf4242bf1d98249ee65223460c5a WatchSource:0}: Error finding container 3a3b4e44997119135b104ebe5a22290525b7bf4242bf1d98249ee65223460c5a: Status 404 returned error can't find the container with id 3a3b4e44997119135b104ebe5a22290525b7bf4242bf1d98249ee65223460c5a
Jul 17 22:06:56.129825 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:56.129742    2563 generic.go:356] "Generic (PLEG): container finished" podID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerID="31db861116aca2193eb18dc32da0bf5b8599f0a4f387d9fd2cd5ce2f65d4dfb4" exitCode=6
Jul 17 22:06:56.129985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:56.129822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" event={"ID":"83f4a191-f64a-4583-8590-c13857cfbc9d","Type":"ContainerDied","Data":"31db861116aca2193eb18dc32da0bf5b8599f0a4f387d9fd2cd5ce2f65d4dfb4"}
Jul 17 22:06:56.129985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:56.129902    2563 scope.go:117] "RemoveContainer" containerID="280f82df7d6a2d1ba8f0c643c79920e3e6cbb8a39f80f542aedb6e2f4c7856f5"
Jul 17 22:06:56.130210 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:56.130190    2563 scope.go:117] "RemoveContainer" containerID="31db861116aca2193eb18dc32da0bf5b8599f0a4f387d9fd2cd5ce2f65d4dfb4"
Jul 17 22:06:56.130485 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:56.130452    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt_odh-ai-gateway-infra(83f4a191-f64a-4583-8590-c13857cfbc9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d"
Jul 17 22:06:56.131063 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:56.131012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" event={"ID":"64013b03-317c-4575-82bc-52f333057f11","Type":"ContainerStarted","Data":"3a3b4e44997119135b104ebe5a22290525b7bf4242bf1d98249ee65223460c5a"}
Jul 17 22:06:56.152824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:56.152800    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"]
Jul 17 22:06:57.139752 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.139720    2563 generic.go:356] "Generic (PLEG): container finished" podID="64013b03-317c-4575-82bc-52f333057f11" containerID="eb1bd0c408376ff8086f1cbb7c1ad2b174f3d279721f143dfa646044063d0288" exitCode=6
Jul 17 22:06:57.140143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.139811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" event={"ID":"64013b03-317c-4575-82bc-52f333057f11","Type":"ContainerDied","Data":"eb1bd0c408376ff8086f1cbb7c1ad2b174f3d279721f143dfa646044063d0288"}
Jul 17 22:06:57.140143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.139993    2563 scope.go:117] "RemoveContainer" containerID="eb1bd0c408376ff8086f1cbb7c1ad2b174f3d279721f143dfa646044063d0288"
Jul 17 22:06:57.286979 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.286953    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:57.378541 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.378517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f4a191-f64a-4583-8590-c13857cfbc9d-maas-api-service-ca\") pod \"83f4a191-f64a-4583-8590-c13857cfbc9d\" (UID: \"83f4a191-f64a-4583-8590-c13857cfbc9d\") "
Jul 17 22:06:57.378926 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.378897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f4a191-f64a-4583-8590-c13857cfbc9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83f4a191-f64a-4583-8590-c13857cfbc9d" (UID: "83f4a191-f64a-4583-8590-c13857cfbc9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:57.480295 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.480268    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f4a191-f64a-4583-8590-c13857cfbc9d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:06:57.579712 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.579673    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"]
Jul 17 22:06:57.580040 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.580028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:06:57.580084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.580042    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:06:57.580084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.580053    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:06:57.580084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.580060    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:06:57.580177 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.580127    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:06:57.580177 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.580135    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:06:57.583267 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.583250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:06:57.590855 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.590833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"]
Jul 17 22:06:57.682100 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.682059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0d25a-fa04-413c-a100-083c7ac67aec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8\" (UID: \"b5f0d25a-fa04-413c-a100-083c7ac67aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:06:57.782504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.782476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0d25a-fa04-413c-a100-083c7ac67aec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8\" (UID: \"b5f0d25a-fa04-413c-a100-083c7ac67aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:06:57.783126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.783107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0d25a-fa04-413c-a100-083c7ac67aec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8\" (UID: \"b5f0d25a-fa04-413c-a100-083c7ac67aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:06:57.894709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:57.894670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:06:58.026208 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.026166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"]
Jul 17 22:06:58.028898 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:06:58.028871    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f0d25a_fa04_413c_a100_083c7ac67aec.slice/crio-2fd8af66d8c018fb30971502c6771e22f29a250f0ea1193771228de4a6e8d162 WatchSource:0}: Error finding container 2fd8af66d8c018fb30971502c6771e22f29a250f0ea1193771228de4a6e8d162: Status 404 returned error can't find the container with id 2fd8af66d8c018fb30971502c6771e22f29a250f0ea1193771228de4a6e8d162
Jul 17 22:06:58.145751 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.145717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" event={"ID":"b5f0d25a-fa04-413c-a100-083c7ac67aec","Type":"ContainerStarted","Data":"2fd8af66d8c018fb30971502c6771e22f29a250f0ea1193771228de4a6e8d162"}
Jul 17 22:06:58.147027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.146998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt" event={"ID":"83f4a191-f64a-4583-8590-c13857cfbc9d","Type":"ContainerDied","Data":"e3cd87993ed47c66c606dddfc96d818274b96d34148b18b787e7d1fe34dac7d7"}
Jul 17 22:06:58.147143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.147053    2563 scope.go:117] "RemoveContainer" containerID="31db861116aca2193eb18dc32da0bf5b8599f0a4f387d9fd2cd5ce2f65d4dfb4"
Jul 17 22:06:58.147143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.147007    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"
Jul 17 22:06:58.148979 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.148954    2563 generic.go:356] "Generic (PLEG): container finished" podID="64013b03-317c-4575-82bc-52f333057f11" containerID="44a5a605a90fa30c70e9032df226cf961b5ed0a3f4c50e0979c9dc4884c2367d" exitCode=6
Jul 17 22:06:58.149074 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.149042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" event={"ID":"64013b03-317c-4575-82bc-52f333057f11","Type":"ContainerDied","Data":"44a5a605a90fa30c70e9032df226cf961b5ed0a3f4c50e0979c9dc4884c2367d"}
Jul 17 22:06:58.149308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.149292    2563 scope.go:117] "RemoveContainer" containerID="44a5a605a90fa30c70e9032df226cf961b5ed0a3f4c50e0979c9dc4884c2367d"
Jul 17 22:06:58.149538 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:58.149518    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr_odh-ai-gateway-infra(64013b03-317c-4575-82bc-52f333057f11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" podUID="64013b03-317c-4575-82bc-52f333057f11"
Jul 17 22:06:58.155560 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.155541    2563 scope.go:117] "RemoveContainer" containerID="eb1bd0c408376ff8086f1cbb7c1ad2b174f3d279721f143dfa646044063d0288"
Jul 17 22:06:58.191143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.191096    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"]
Jul 17 22:06:58.192968 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.192946    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fvdwt"]
Jul 17 22:06:58.603638 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:58.603580    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" path="/var/lib/kubelet/pods/83f4a191-f64a-4583-8590-c13857cfbc9d/volumes"
Jul 17 22:06:59.155095 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:59.155062    2563 generic.go:356] "Generic (PLEG): container finished" podID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerID="a7a88325c2ee154a05392cba3d48ee039c8a0b6aecd409292923429f84aab869" exitCode=6
Jul 17 22:06:59.155520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:59.155140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" event={"ID":"b5f0d25a-fa04-413c-a100-083c7ac67aec","Type":"ContainerDied","Data":"a7a88325c2ee154a05392cba3d48ee039c8a0b6aecd409292923429f84aab869"}
Jul 17 22:06:59.155520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:59.155330    2563 scope.go:117] "RemoveContainer" containerID="a7a88325c2ee154a05392cba3d48ee039c8a0b6aecd409292923429f84aab869"
Jul 17 22:06:59.157892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:06:59.157878    2563 scope.go:117] "RemoveContainer" containerID="44a5a605a90fa30c70e9032df226cf961b5ed0a3f4c50e0979c9dc4884c2367d"
Jul 17 22:06:59.158049 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:06:59.158033    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr_odh-ai-gateway-infra(64013b03-317c-4575-82bc-52f333057f11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" podUID="64013b03-317c-4575-82bc-52f333057f11"
Jul 17 22:07:00.164801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:00.164770    2563 generic.go:356] "Generic (PLEG): container finished" podID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerID="26f2fd06700dd2be006f684aa5424a4c614a63ea6d838a15a3cbe1befab9dc84" exitCode=6
Jul 17 22:07:00.165265 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:00.164857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" event={"ID":"b5f0d25a-fa04-413c-a100-083c7ac67aec","Type":"ContainerDied","Data":"26f2fd06700dd2be006f684aa5424a4c614a63ea6d838a15a3cbe1befab9dc84"}
Jul 17 22:07:00.165265 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:00.164916    2563 scope.go:117] "RemoveContainer" containerID="a7a88325c2ee154a05392cba3d48ee039c8a0b6aecd409292923429f84aab869"
Jul 17 22:07:00.165265 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:00.165019    2563 scope.go:117] "RemoveContainer" containerID="26f2fd06700dd2be006f684aa5424a4c614a63ea6d838a15a3cbe1befab9dc84"
Jul 17 22:07:00.165456 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:00.165283    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8_odh-ai-gateway-infra(b5f0d25a-fa04-413c-a100-083c7ac67aec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec"
Jul 17 22:07:01.171674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:01.171649    2563 scope.go:117] "RemoveContainer" containerID="26f2fd06700dd2be006f684aa5424a4c614a63ea6d838a15a3cbe1befab9dc84"
Jul 17 22:07:01.172058 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:01.171834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8_odh-ai-gateway-infra(b5f0d25a-fa04-413c-a100-083c7ac67aec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec"
Jul 17 22:07:11.597405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:11.597368    2563 scope.go:117] "RemoveContainer" containerID="44a5a605a90fa30c70e9032df226cf961b5ed0a3f4c50e0979c9dc4884c2367d"
Jul 17 22:07:12.231016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.230985    2563 generic.go:356] "Generic (PLEG): container finished" podID="64013b03-317c-4575-82bc-52f333057f11" containerID="a3e08550b165260a859b61eff49190a8a130bba46dc6cb1f340b47d2ce114752" exitCode=6
Jul 17 22:07:12.231170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.231056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" event={"ID":"64013b03-317c-4575-82bc-52f333057f11","Type":"ContainerDied","Data":"a3e08550b165260a859b61eff49190a8a130bba46dc6cb1f340b47d2ce114752"}
Jul 17 22:07:12.231170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.231126    2563 scope.go:117] "RemoveContainer" containerID="44a5a605a90fa30c70e9032df226cf961b5ed0a3f4c50e0979c9dc4884c2367d"
Jul 17 22:07:12.231364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.231351    2563 scope.go:117] "RemoveContainer" containerID="a3e08550b165260a859b61eff49190a8a130bba46dc6cb1f340b47d2ce114752"
Jul 17 22:07:12.231605 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:12.231567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr_odh-ai-gateway-infra(64013b03-317c-4575-82bc-52f333057f11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" podUID="64013b03-317c-4575-82bc-52f333057f11"
Jul 17 22:07:12.941960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.941925    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"]
Jul 17 22:07:12.942347 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.942331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:07:12.942347 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.942343    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:07:12.942435 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.942424    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f4a191-f64a-4583-8590-c13857cfbc9d" containerName="revoke-keys"
Jul 17 22:07:12.946642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.946626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:12.955138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:12.955112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"]
Jul 17 22:07:13.017646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.017579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a90583d-7408-42d3-b422-a4cc2d64fbb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg\" (UID: \"4a90583d-7408-42d3-b422-a4cc2d64fbb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:13.118843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.118794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a90583d-7408-42d3-b422-a4cc2d64fbb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg\" (UID: \"4a90583d-7408-42d3-b422-a4cc2d64fbb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:13.119420 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.119397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a90583d-7408-42d3-b422-a4cc2d64fbb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg\" (UID: \"4a90583d-7408-42d3-b422-a4cc2d64fbb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:13.255777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.255692    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"]
Jul 17 22:07:13.258296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.258273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:13.401341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.401314    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:07:13.408180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.408159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"]
Jul 17 22:07:13.410731 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:07:13.410704    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a90583d_7408_42d3_b422_a4cc2d64fbb0.slice/crio-dfcc1d74aeaff9489ec5c9993b5646e02605a1abed5c75074e95f26fd352ef59 WatchSource:0}: Error finding container dfcc1d74aeaff9489ec5c9993b5646e02605a1abed5c75074e95f26fd352ef59: Status 404 returned error can't find the container with id dfcc1d74aeaff9489ec5c9993b5646e02605a1abed5c75074e95f26fd352ef59
Jul 17 22:07:13.522465 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.522386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64013b03-317c-4575-82bc-52f333057f11-maas-api-service-ca\") pod \"64013b03-317c-4575-82bc-52f333057f11\" (UID: \"64013b03-317c-4575-82bc-52f333057f11\") "
Jul 17 22:07:13.522762 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.522739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64013b03-317c-4575-82bc-52f333057f11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64013b03-317c-4575-82bc-52f333057f11" (UID: "64013b03-317c-4575-82bc-52f333057f11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:13.623880 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:13.623847    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64013b03-317c-4575-82bc-52f333057f11-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.244189 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.244157    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"
Jul 17 22:07:14.244189 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.244174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr" event={"ID":"64013b03-317c-4575-82bc-52f333057f11","Type":"ContainerDied","Data":"3a3b4e44997119135b104ebe5a22290525b7bf4242bf1d98249ee65223460c5a"}
Jul 17 22:07:14.244719 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.244242    2563 scope.go:117] "RemoveContainer" containerID="a3e08550b165260a859b61eff49190a8a130bba46dc6cb1f340b47d2ce114752"
Jul 17 22:07:14.245942 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.245918    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerID="7f4461b56c6be3f92c28b2cb16befad31a44263fed63829804bf0f16499b0945" exitCode=7
Jul 17 22:07:14.246090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.245989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" event={"ID":"4a90583d-7408-42d3-b422-a4cc2d64fbb0","Type":"ContainerDied","Data":"7f4461b56c6be3f92c28b2cb16befad31a44263fed63829804bf0f16499b0945"}
Jul 17 22:07:14.246090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.246055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" event={"ID":"4a90583d-7408-42d3-b422-a4cc2d64fbb0","Type":"ContainerStarted","Data":"dfcc1d74aeaff9489ec5c9993b5646e02605a1abed5c75074e95f26fd352ef59"}
Jul 17 22:07:14.246212 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.246165    2563 scope.go:117] "RemoveContainer" containerID="7f4461b56c6be3f92c28b2cb16befad31a44263fed63829804bf0f16499b0945"
Jul 17 22:07:14.275864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.275838    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"]
Jul 17 22:07:14.282116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.282095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t76cr"]
Jul 17 22:07:14.602645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.602614    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64013b03-317c-4575-82bc-52f333057f11" path="/var/lib/kubelet/pods/64013b03-317c-4575-82bc-52f333057f11/volumes"
Jul 17 22:07:14.603212 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:14.603198    2563 scope.go:117] "RemoveContainer" containerID="26f2fd06700dd2be006f684aa5424a4c614a63ea6d838a15a3cbe1befab9dc84"
Jul 17 22:07:15.253373 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.253281    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerID="cf3de9099ada255e1712764eda6543a298afa0d31f8620b72f7e20af6366cdbb" exitCode=7
Jul 17 22:07:15.253840 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.253374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" event={"ID":"4a90583d-7408-42d3-b422-a4cc2d64fbb0","Type":"ContainerDied","Data":"cf3de9099ada255e1712764eda6543a298afa0d31f8620b72f7e20af6366cdbb"}
Jul 17 22:07:15.253840 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.253443    2563 scope.go:117] "RemoveContainer" containerID="7f4461b56c6be3f92c28b2cb16befad31a44263fed63829804bf0f16499b0945"
Jul 17 22:07:15.253840 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.253662    2563 scope.go:117] "RemoveContainer" containerID="cf3de9099ada255e1712764eda6543a298afa0d31f8620b72f7e20af6366cdbb"
Jul 17 22:07:15.253994 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:15.253912    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg_odh-ai-gateway-infra(4a90583d-7408-42d3-b422-a4cc2d64fbb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0"
Jul 17 22:07:15.255350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.255331    2563 generic.go:356] "Generic (PLEG): container finished" podID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerID="68474d22b09b885c04a899fb5089285507589a4f4399b4a6c067d02b021a2d80" exitCode=6
Jul 17 22:07:15.255460 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.255379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" event={"ID":"b5f0d25a-fa04-413c-a100-083c7ac67aec","Type":"ContainerDied","Data":"68474d22b09b885c04a899fb5089285507589a4f4399b4a6c067d02b021a2d80"}
Jul 17 22:07:15.255708 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.255693    2563 scope.go:117] "RemoveContainer" containerID="68474d22b09b885c04a899fb5089285507589a4f4399b4a6c067d02b021a2d80"
Jul 17 22:07:15.255892 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:15.255872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8_odh-ai-gateway-infra(b5f0d25a-fa04-413c-a100-083c7ac67aec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec"
Jul 17 22:07:15.264882 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:15.264698    2563 scope.go:117] "RemoveContainer" containerID="26f2fd06700dd2be006f684aa5424a4c614a63ea6d838a15a3cbe1befab9dc84"
Jul 17 22:07:16.267701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.267677    2563 scope.go:117] "RemoveContainer" containerID="cf3de9099ada255e1712764eda6543a298afa0d31f8620b72f7e20af6366cdbb"
Jul 17 22:07:16.268118 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:16.267887    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg_odh-ai-gateway-infra(4a90583d-7408-42d3-b422-a4cc2d64fbb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0"
Jul 17 22:07:16.298980 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.298950    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"]
Jul 17 22:07:16.439726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.439704    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:07:16.553171 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.553086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0d25a-fa04-413c-a100-083c7ac67aec-maas-api-service-ca\") pod \"b5f0d25a-fa04-413c-a100-083c7ac67aec\" (UID: \"b5f0d25a-fa04-413c-a100-083c7ac67aec\") "
Jul 17 22:07:16.553636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.553589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f0d25a-fa04-413c-a100-083c7ac67aec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5f0d25a-fa04-413c-a100-083c7ac67aec" (UID: "b5f0d25a-fa04-413c-a100-083c7ac67aec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:16.654905 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.654872    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0d25a-fa04-413c-a100-083c7ac67aec-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:07:16.723136 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.723107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:07:16.728615 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:16.728581    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:07:17.272983 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:17.272903    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"
Jul 17 22:07:17.273392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:17.272902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8" event={"ID":"b5f0d25a-fa04-413c-a100-083c7ac67aec","Type":"ContainerDied","Data":"2fd8af66d8c018fb30971502c6771e22f29a250f0ea1193771228de4a6e8d162"}
Jul 17 22:07:17.273392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:17.273044    2563 scope.go:117] "RemoveContainer" containerID="68474d22b09b885c04a899fb5089285507589a4f4399b4a6c067d02b021a2d80"
Jul 17 22:07:17.289091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:17.289064    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"]
Jul 17 22:07:17.293272 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:17.293250    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-grxc8"]
Jul 17 22:07:18.603756 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:18.603716    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" path="/var/lib/kubelet/pods/b5f0d25a-fa04-413c-a100-083c7ac67aec/volumes"
Jul 17 22:07:28.597427 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:28.597345    2563 scope.go:117] "RemoveContainer" containerID="cf3de9099ada255e1712764eda6543a298afa0d31f8620b72f7e20af6366cdbb"
Jul 17 22:07:29.337339 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:29.337309    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerID="3fc2c9c45f1f451efa5f8d2d7ef5c18d1653b803e744134aa412142538a9098d" exitCode=6
Jul 17 22:07:29.337504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:29.337381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" event={"ID":"4a90583d-7408-42d3-b422-a4cc2d64fbb0","Type":"ContainerDied","Data":"3fc2c9c45f1f451efa5f8d2d7ef5c18d1653b803e744134aa412142538a9098d"}
Jul 17 22:07:29.337504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:29.337439    2563 scope.go:117] "RemoveContainer" containerID="cf3de9099ada255e1712764eda6543a298afa0d31f8620b72f7e20af6366cdbb"
Jul 17 22:07:29.337785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:29.337766    2563 scope.go:117] "RemoveContainer" containerID="3fc2c9c45f1f451efa5f8d2d7ef5c18d1653b803e744134aa412142538a9098d"
Jul 17 22:07:29.338032 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:29.338000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg_odh-ai-gateway-infra(4a90583d-7408-42d3-b422-a4cc2d64fbb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0"
Jul 17 22:07:30.363535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:30.363500    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"]
Jul 17 22:07:30.506309 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:30.506287    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:30.678442 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:30.678414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a90583d-7408-42d3-b422-a4cc2d64fbb0-maas-api-service-ca\") pod \"4a90583d-7408-42d3-b422-a4cc2d64fbb0\" (UID: \"4a90583d-7408-42d3-b422-a4cc2d64fbb0\") "
Jul 17 22:07:30.678878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:30.678854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a90583d-7408-42d3-b422-a4cc2d64fbb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a90583d-7408-42d3-b422-a4cc2d64fbb0" (UID: "4a90583d-7408-42d3-b422-a4cc2d64fbb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:30.779978 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:30.779941    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a90583d-7408-42d3-b422-a4cc2d64fbb0-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:07:31.349861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:31.349833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"
Jul 17 22:07:31.350039 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:31.349833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg" event={"ID":"4a90583d-7408-42d3-b422-a4cc2d64fbb0","Type":"ContainerDied","Data":"dfcc1d74aeaff9489ec5c9993b5646e02605a1abed5c75074e95f26fd352ef59"}
Jul 17 22:07:31.350039 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:31.349970    2563 scope.go:117] "RemoveContainer" containerID="3fc2c9c45f1f451efa5f8d2d7ef5c18d1653b803e744134aa412142538a9098d"
Jul 17 22:07:31.374845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:31.374818    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"]
Jul 17 22:07:31.380372 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:31.380344    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-xrbjg"]
Jul 17 22:07:32.602935 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:32.602902    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" path="/var/lib/kubelet/pods/4a90583d-7408-42d3-b422-a4cc2d64fbb0/volumes"
Jul 17 22:07:45.534557 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534512    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"]
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534913    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534930    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534939    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534944    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534953    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534959    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534977    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534982    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.534997    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535006    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:07:45.535010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535020    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535075    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535087    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535096    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535102    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535108    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535112    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:07:45.535348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.535118    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:07:45.539580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.539563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:07:45.542330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.542292    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:07:45.544944 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.544854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"]
Jul 17 22:07:45.607642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.607580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534d4a9d-083e-40d5-a485-41e54a9b808d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s\" (UID: \"534d4a9d-083e-40d5-a485-41e54a9b808d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:07:45.708358 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.708322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534d4a9d-083e-40d5-a485-41e54a9b808d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s\" (UID: \"534d4a9d-083e-40d5-a485-41e54a9b808d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:07:45.708983 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.708965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534d4a9d-083e-40d5-a485-41e54a9b808d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s\" (UID: \"534d4a9d-083e-40d5-a485-41e54a9b808d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:07:45.852090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.852008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:07:45.992482 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:45.992455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"]
Jul 17 22:07:45.994302 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:07:45.994274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534d4a9d_083e_40d5_a485_41e54a9b808d.slice/crio-9f9ef388145fe3813236b7bc889b976ad29ab97150fc0a1e6369b844a1f58d9f WatchSource:0}: Error finding container 9f9ef388145fe3813236b7bc889b976ad29ab97150fc0a1e6369b844a1f58d9f: Status 404 returned error can't find the container with id 9f9ef388145fe3813236b7bc889b976ad29ab97150fc0a1e6369b844a1f58d9f
Jul 17 22:07:46.422107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:46.422019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" event={"ID":"534d4a9d-083e-40d5-a485-41e54a9b808d","Type":"ContainerStarted","Data":"820f7b7d94db464e2b9491526f50b55054406425149d50534c30f6031146bdd5"}
Jul 17 22:07:46.422107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:46.422077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" event={"ID":"534d4a9d-083e-40d5-a485-41e54a9b808d","Type":"ContainerStarted","Data":"9f9ef388145fe3813236b7bc889b976ad29ab97150fc0a1e6369b844a1f58d9f"}
Jul 17 22:07:46.438536 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:46.438482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" podStartSLOduration=1.4384668600000001 podStartE2EDuration="1.43846686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:46.437828262 +0000 UTC m=+2430.350068551" watchObservedRunningTime="2026-07-17 22:07:46.43846686 +0000 UTC m=+2430.350707154"
Jul 17 22:07:47.427903 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:47.427868    2563 generic.go:356] "Generic (PLEG): container finished" podID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerID="820f7b7d94db464e2b9491526f50b55054406425149d50534c30f6031146bdd5" exitCode=6
Jul 17 22:07:47.428407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:47.427913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" event={"ID":"534d4a9d-083e-40d5-a485-41e54a9b808d","Type":"ContainerDied","Data":"820f7b7d94db464e2b9491526f50b55054406425149d50534c30f6031146bdd5"}
Jul 17 22:07:47.428407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:47.428198    2563 scope.go:117] "RemoveContainer" containerID="820f7b7d94db464e2b9491526f50b55054406425149d50534c30f6031146bdd5"
Jul 17 22:07:48.434460 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:48.434428    2563 generic.go:356] "Generic (PLEG): container finished" podID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerID="5ca4feeb966997cbfc07c68d918aafa3a2ab25d59db44e7699e529d0f6ba4d7e" exitCode=6
Jul 17 22:07:48.434899 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:48.434477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" event={"ID":"534d4a9d-083e-40d5-a485-41e54a9b808d","Type":"ContainerDied","Data":"5ca4feeb966997cbfc07c68d918aafa3a2ab25d59db44e7699e529d0f6ba4d7e"}
Jul 17 22:07:48.434899 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:48.434550    2563 scope.go:117] "RemoveContainer" containerID="820f7b7d94db464e2b9491526f50b55054406425149d50534c30f6031146bdd5"
Jul 17 22:07:48.434899 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:48.434748    2563 scope.go:117] "RemoveContainer" containerID="5ca4feeb966997cbfc07c68d918aafa3a2ab25d59db44e7699e529d0f6ba4d7e"
Jul 17 22:07:48.435049 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:48.434965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s_odh-ai-gateway-infra(534d4a9d-083e-40d5-a485-41e54a9b808d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d"
Jul 17 22:07:49.440875 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:07:49.440842    2563 scope.go:117] "RemoveContainer" containerID="5ca4feeb966997cbfc07c68d918aafa3a2ab25d59db44e7699e529d0f6ba4d7e"
Jul 17 22:07:49.441273 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:07:49.441022    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s_odh-ai-gateway-infra(534d4a9d-083e-40d5-a485-41e54a9b808d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d"
Jul 17 22:08:02.980625 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.980561    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"]
Jul 17 22:08:02.981224 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981144    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:08:02.981224 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981164    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f0d25a-fa04-413c-a100-083c7ac67aec" containerName="revoke-keys"
Jul 17 22:08:02.981224 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:08:02.981224 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981199    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:08:02.981224 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981209    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:08:02.981224 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:08:02.981544 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981303    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64013b03-317c-4575-82bc-52f333057f11" containerName="revoke-keys"
Jul 17 22:08:02.981544 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.981322    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a90583d-7408-42d3-b422-a4cc2d64fbb0" containerName="revoke-keys"
Jul 17 22:08:02.984434 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.984413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:02.992590 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:02.992564    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"]
Jul 17 22:08:03.066655 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:03.066617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60167219-53ac-4b8b-a92a-73cd5a517dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7\" (UID: \"60167219-53ac-4b8b-a92a-73cd5a517dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:03.167951 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:03.167918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60167219-53ac-4b8b-a92a-73cd5a517dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7\" (UID: \"60167219-53ac-4b8b-a92a-73cd5a517dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:03.168649 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:03.168628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60167219-53ac-4b8b-a92a-73cd5a517dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7\" (UID: \"60167219-53ac-4b8b-a92a-73cd5a517dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:03.297123 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:03.297041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:03.596760 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:03.596677    2563 scope.go:117] "RemoveContainer" containerID="5ca4feeb966997cbfc07c68d918aafa3a2ab25d59db44e7699e529d0f6ba4d7e"
Jul 17 22:08:03.643522 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:03.643500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"]
Jul 17 22:08:03.645248 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:08:03.645211    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60167219_53ac_4b8b_a92a_73cd5a517dc2.slice/crio-c8cd4dc7d7bca26a11369c5756ad33283d504973cb8addc6303a279960e07ad3 WatchSource:0}: Error finding container c8cd4dc7d7bca26a11369c5756ad33283d504973cb8addc6303a279960e07ad3: Status 404 returned error can't find the container with id c8cd4dc7d7bca26a11369c5756ad33283d504973cb8addc6303a279960e07ad3
Jul 17 22:08:04.511549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.511445    2563 generic.go:356] "Generic (PLEG): container finished" podID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerID="ccc9ad68b3a8558d683eccee2d4df3344f8a0699cfad9c96a37e06527ad6769d" exitCode=6
Jul 17 22:08:04.511549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.511523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" event={"ID":"534d4a9d-083e-40d5-a485-41e54a9b808d","Type":"ContainerDied","Data":"ccc9ad68b3a8558d683eccee2d4df3344f8a0699cfad9c96a37e06527ad6769d"}
Jul 17 22:08:04.512104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.511648    2563 scope.go:117] "RemoveContainer" containerID="5ca4feeb966997cbfc07c68d918aafa3a2ab25d59db44e7699e529d0f6ba4d7e"
Jul 17 22:08:04.512104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.511996    2563 scope.go:117] "RemoveContainer" containerID="ccc9ad68b3a8558d683eccee2d4df3344f8a0699cfad9c96a37e06527ad6769d"
Jul 17 22:08:04.512231 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:04.512207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s_odh-ai-gateway-infra(534d4a9d-083e-40d5-a485-41e54a9b808d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d"
Jul 17 22:08:04.513131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.513110    2563 generic.go:356] "Generic (PLEG): container finished" podID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerID="2d4665ee07b58a80e1ed8de6686a385ee696f1599545d993920807880fa527b7" exitCode=6
Jul 17 22:08:04.513239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.513164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" event={"ID":"60167219-53ac-4b8b-a92a-73cd5a517dc2","Type":"ContainerDied","Data":"2d4665ee07b58a80e1ed8de6686a385ee696f1599545d993920807880fa527b7"}
Jul 17 22:08:04.513239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.513199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" event={"ID":"60167219-53ac-4b8b-a92a-73cd5a517dc2","Type":"ContainerStarted","Data":"c8cd4dc7d7bca26a11369c5756ad33283d504973cb8addc6303a279960e07ad3"}
Jul 17 22:08:04.513334 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:04.513319    2563 scope.go:117] "RemoveContainer" containerID="2d4665ee07b58a80e1ed8de6686a385ee696f1599545d993920807880fa527b7"
Jul 17 22:08:05.521185 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.521150    2563 generic.go:356] "Generic (PLEG): container finished" podID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerID="cf039af0cf0edf2e77ce8571f5fb2eb935ab1782c112e9eb9dc627c143ab44e6" exitCode=6
Jul 17 22:08:05.521685 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.521228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" event={"ID":"60167219-53ac-4b8b-a92a-73cd5a517dc2","Type":"ContainerDied","Data":"cf039af0cf0edf2e77ce8571f5fb2eb935ab1782c112e9eb9dc627c143ab44e6"}
Jul 17 22:08:05.521685 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.521293    2563 scope.go:117] "RemoveContainer" containerID="2d4665ee07b58a80e1ed8de6686a385ee696f1599545d993920807880fa527b7"
Jul 17 22:08:05.521685 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.521501    2563 scope.go:117] "RemoveContainer" containerID="cf039af0cf0edf2e77ce8571f5fb2eb935ab1782c112e9eb9dc627c143ab44e6"
Jul 17 22:08:05.521850 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:05.521723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7_odh-ai-gateway-infra(60167219-53ac-4b8b-a92a-73cd5a517dc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2"
Jul 17 22:08:05.536090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.536068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"]
Jul 17 22:08:05.677289 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.677267    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:08:05.794177 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.794095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534d4a9d-083e-40d5-a485-41e54a9b808d-maas-api-service-ca\") pod \"534d4a9d-083e-40d5-a485-41e54a9b808d\" (UID: \"534d4a9d-083e-40d5-a485-41e54a9b808d\") "
Jul 17 22:08:05.794433 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.794411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534d4a9d-083e-40d5-a485-41e54a9b808d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "534d4a9d-083e-40d5-a485-41e54a9b808d" (UID: "534d4a9d-083e-40d5-a485-41e54a9b808d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:05.895631 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:05.895580    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534d4a9d-083e-40d5-a485-41e54a9b808d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:08:06.527715 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.527687    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"
Jul 17 22:08:06.528182 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.527686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s" event={"ID":"534d4a9d-083e-40d5-a485-41e54a9b808d","Type":"ContainerDied","Data":"9f9ef388145fe3813236b7bc889b976ad29ab97150fc0a1e6369b844a1f58d9f"}
Jul 17 22:08:06.528182 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.527817    2563 scope.go:117] "RemoveContainer" containerID="ccc9ad68b3a8558d683eccee2d4df3344f8a0699cfad9c96a37e06527ad6769d"
Jul 17 22:08:06.529442 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.529423    2563 scope.go:117] "RemoveContainer" containerID="cf039af0cf0edf2e77ce8571f5fb2eb935ab1782c112e9eb9dc627c143ab44e6"
Jul 17 22:08:06.529654 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:06.529634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7_odh-ai-gateway-infra(60167219-53ac-4b8b-a92a-73cd5a517dc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2"
Jul 17 22:08:06.558343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.558313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"]
Jul 17 22:08:06.562570 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.562549    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-qdj9s"]
Jul 17 22:08:06.602400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:06.602371    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" path="/var/lib/kubelet/pods/534d4a9d-083e-40d5-a485-41e54a9b808d/volumes"
Jul 17 22:08:18.602426 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:18.602398    2563 scope.go:117] "RemoveContainer" containerID="cf039af0cf0edf2e77ce8571f5fb2eb935ab1782c112e9eb9dc627c143ab44e6"
Jul 17 22:08:19.593581 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.593489    2563 generic.go:356] "Generic (PLEG): container finished" podID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerID="1c332bd4d2c6bac0846b874c5560e6dd900b5cdb7c6d1bca930705cf7365fc10" exitCode=6
Jul 17 22:08:19.593581 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.593566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" event={"ID":"60167219-53ac-4b8b-a92a-73cd5a517dc2","Type":"ContainerDied","Data":"1c332bd4d2c6bac0846b874c5560e6dd900b5cdb7c6d1bca930705cf7365fc10"}
Jul 17 22:08:19.593783 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.593647    2563 scope.go:117] "RemoveContainer" containerID="cf039af0cf0edf2e77ce8571f5fb2eb935ab1782c112e9eb9dc627c143ab44e6"
Jul 17 22:08:19.593937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.593922    2563 scope.go:117] "RemoveContainer" containerID="1c332bd4d2c6bac0846b874c5560e6dd900b5cdb7c6d1bca930705cf7365fc10"
Jul 17 22:08:19.594237 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:19.594198    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7_odh-ai-gateway-infra(60167219-53ac-4b8b-a92a-73cd5a517dc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2"
Jul 17 22:08:19.836107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836073    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"]
Jul 17 22:08:19.836494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.836494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836450    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.836494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.836494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836474    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.836679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836530    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.836679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836539    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.836679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.836546    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:19.840643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.840623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:19.853027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:19.853002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"]
Jul 17 22:08:20.028116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.028082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98ec0e4-0596-4d8b-863c-7ee2202156ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp\" (UID: \"a98ec0e4-0596-4d8b-863c-7ee2202156ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:20.129675 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.129542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98ec0e4-0596-4d8b-863c-7ee2202156ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp\" (UID: \"a98ec0e4-0596-4d8b-863c-7ee2202156ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:20.130181 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.130158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98ec0e4-0596-4d8b-863c-7ee2202156ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp\" (UID: \"a98ec0e4-0596-4d8b-863c-7ee2202156ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:20.156577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.156553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:20.292485 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.292455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"]
Jul 17 22:08:20.294661 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:08:20.294628    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98ec0e4_0596_4d8b_863c_7ee2202156ed.slice/crio-9843c118d0daa31764527d2efe69466c92ed636ca89aaf3fe31b9a565d189f38 WatchSource:0}: Error finding container 9843c118d0daa31764527d2efe69466c92ed636ca89aaf3fe31b9a565d189f38: Status 404 returned error can't find the container with id 9843c118d0daa31764527d2efe69466c92ed636ca89aaf3fe31b9a565d189f38
Jul 17 22:08:20.602585 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.602544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" event={"ID":"a98ec0e4-0596-4d8b-863c-7ee2202156ed","Type":"ContainerStarted","Data":"4cbe467c24e13aaddbe5096857a45d679d9b590e1b23f8c1885eb3bc13e74c49"}
Jul 17 22:08:20.602774 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.602623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" event={"ID":"a98ec0e4-0596-4d8b-863c-7ee2202156ed","Type":"ContainerStarted","Data":"9843c118d0daa31764527d2efe69466c92ed636ca89aaf3fe31b9a565d189f38"}
Jul 17 22:08:20.615410 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.615365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" podStartSLOduration=1.615349117 podStartE2EDuration="1.615349117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:20.614694562 +0000 UTC m=+2464.526934841" watchObservedRunningTime="2026-07-17 22:08:20.615349117 +0000 UTC m=+2464.527589409"
Jul 17 22:08:20.629562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.629534    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"]
Jul 17 22:08:20.769697 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.769674    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:20.936308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.936279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60167219-53ac-4b8b-a92a-73cd5a517dc2-maas-api-service-ca\") pod \"60167219-53ac-4b8b-a92a-73cd5a517dc2\" (UID: \"60167219-53ac-4b8b-a92a-73cd5a517dc2\") "
Jul 17 22:08:20.937006 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:20.936576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60167219-53ac-4b8b-a92a-73cd5a517dc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60167219-53ac-4b8b-a92a-73cd5a517dc2" (UID: "60167219-53ac-4b8b-a92a-73cd5a517dc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:21.037496 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.037412    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60167219-53ac-4b8b-a92a-73cd5a517dc2-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:08:21.608002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.607967    2563 generic.go:356] "Generic (PLEG): container finished" podID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerID="4cbe467c24e13aaddbe5096857a45d679d9b590e1b23f8c1885eb3bc13e74c49" exitCode=6
Jul 17 22:08:21.608195 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.608050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" event={"ID":"a98ec0e4-0596-4d8b-863c-7ee2202156ed","Type":"ContainerDied","Data":"4cbe467c24e13aaddbe5096857a45d679d9b590e1b23f8c1885eb3bc13e74c49"}
Jul 17 22:08:21.608326 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.608308    2563 scope.go:117] "RemoveContainer" containerID="4cbe467c24e13aaddbe5096857a45d679d9b590e1b23f8c1885eb3bc13e74c49"
Jul 17 22:08:21.609384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.609362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7" event={"ID":"60167219-53ac-4b8b-a92a-73cd5a517dc2","Type":"ContainerDied","Data":"c8cd4dc7d7bca26a11369c5756ad33283d504973cb8addc6303a279960e07ad3"}
Jul 17 22:08:21.609384 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.609380    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"
Jul 17 22:08:21.609544 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.609411    2563 scope.go:117] "RemoveContainer" containerID="1c332bd4d2c6bac0846b874c5560e6dd900b5cdb7c6d1bca930705cf7365fc10"
Jul 17 22:08:21.635165 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.635138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"]
Jul 17 22:08:21.641161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:21.641126    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-hzsx7"]
Jul 17 22:08:22.602862 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:22.602830    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" path="/var/lib/kubelet/pods/60167219-53ac-4b8b-a92a-73cd5a517dc2/volumes"
Jul 17 22:08:22.616010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:22.615982    2563 generic.go:356] "Generic (PLEG): container finished" podID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerID="1b00ae7786d2a40b271c6fda59508f5051bf182f95cbc68eb101f349cb75add2" exitCode=6
Jul 17 22:08:22.616159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:22.616065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" event={"ID":"a98ec0e4-0596-4d8b-863c-7ee2202156ed","Type":"ContainerDied","Data":"1b00ae7786d2a40b271c6fda59508f5051bf182f95cbc68eb101f349cb75add2"}
Jul 17 22:08:22.616159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:22.616132    2563 scope.go:117] "RemoveContainer" containerID="4cbe467c24e13aaddbe5096857a45d679d9b590e1b23f8c1885eb3bc13e74c49"
Jul 17 22:08:22.616349 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:22.616336    2563 scope.go:117] "RemoveContainer" containerID="1b00ae7786d2a40b271c6fda59508f5051bf182f95cbc68eb101f349cb75add2"
Jul 17 22:08:22.616573 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:22.616552    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp_odh-ai-gateway-infra(a98ec0e4-0596-4d8b-863c-7ee2202156ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed"
Jul 17 22:08:23.624020 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:23.623996    2563 scope.go:117] "RemoveContainer" containerID="1b00ae7786d2a40b271c6fda59508f5051bf182f95cbc68eb101f349cb75add2"
Jul 17 22:08:23.624405 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:23.624161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp_odh-ai-gateway-infra(a98ec0e4-0596-4d8b-863c-7ee2202156ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed"
Jul 17 22:08:36.602481 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:36.602451    2563 scope.go:117] "RemoveContainer" containerID="1b00ae7786d2a40b271c6fda59508f5051bf182f95cbc68eb101f349cb75add2"
Jul 17 22:08:37.690780 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:37.690746    2563 generic.go:356] "Generic (PLEG): container finished" podID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerID="51e164ed489ffcdcbd293742688739f1a73d2203c59c7b7ec89caa83fa89b8d4" exitCode=6
Jul 17 22:08:37.691249 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:37.690795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" event={"ID":"a98ec0e4-0596-4d8b-863c-7ee2202156ed","Type":"ContainerDied","Data":"51e164ed489ffcdcbd293742688739f1a73d2203c59c7b7ec89caa83fa89b8d4"}
Jul 17 22:08:37.691249 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:37.690841    2563 scope.go:117] "RemoveContainer" containerID="1b00ae7786d2a40b271c6fda59508f5051bf182f95cbc68eb101f349cb75add2"
Jul 17 22:08:37.691249 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:37.691150    2563 scope.go:117] "RemoveContainer" containerID="51e164ed489ffcdcbd293742688739f1a73d2203c59c7b7ec89caa83fa89b8d4"
Jul 17 22:08:37.691431 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:37.691397    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp_odh-ai-gateway-infra(a98ec0e4-0596-4d8b-863c-7ee2202156ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed"
Jul 17 22:08:38.716400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:38.716359    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"]
Jul 17 22:08:38.857707 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:38.857682    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:38.875792 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:38.875579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98ec0e4-0596-4d8b-863c-7ee2202156ed-maas-api-service-ca\") pod \"a98ec0e4-0596-4d8b-863c-7ee2202156ed\" (UID: \"a98ec0e4-0596-4d8b-863c-7ee2202156ed\") "
Jul 17 22:08:38.876175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:38.876133    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98ec0e4-0596-4d8b-863c-7ee2202156ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a98ec0e4-0596-4d8b-863c-7ee2202156ed" (UID: "a98ec0e4-0596-4d8b-863c-7ee2202156ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:38.976655 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:38.976545    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98ec0e4-0596-4d8b-863c-7ee2202156ed-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:08:39.701803 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:39.701768    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"
Jul 17 22:08:39.702002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:39.701808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp" event={"ID":"a98ec0e4-0596-4d8b-863c-7ee2202156ed","Type":"ContainerDied","Data":"9843c118d0daa31764527d2efe69466c92ed636ca89aaf3fe31b9a565d189f38"}
Jul 17 22:08:39.702002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:39.701869    2563 scope.go:117] "RemoveContainer" containerID="51e164ed489ffcdcbd293742688739f1a73d2203c59c7b7ec89caa83fa89b8d4"
Jul 17 22:08:39.724624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:39.724584    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"]
Jul 17 22:08:39.732785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:39.732762    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-4r6lp"]
Jul 17 22:08:40.603307 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:40.603273    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" path="/var/lib/kubelet/pods/a98ec0e4-0596-4d8b-863c-7ee2202156ed/volumes"
Jul 17 22:08:53.028563 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028526    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"]
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028911    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="534d4a9d-083e-40d5-a485-41e54a9b808d" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028918    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028923    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028952    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028958    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.028990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.029044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.029052    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:08:53.029506 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.029060    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.029506 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.029066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:08:53.029506 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.029072    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60167219-53ac-4b8b-a92a-73cd5a517dc2" containerName="revoke-keys"
Jul 17 22:08:53.031994 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.031977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:08:53.034480 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.034464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:08:53.038257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.038233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"]
Jul 17 22:08:53.101705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.101674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b383bb9-0516-44ca-9352-d2404e064986-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m\" (UID: \"3b383bb9-0516-44ca-9352-d2404e064986\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:08:53.202338 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.202301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b383bb9-0516-44ca-9352-d2404e064986-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m\" (UID: \"3b383bb9-0516-44ca-9352-d2404e064986\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:08:53.202942 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.202922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b383bb9-0516-44ca-9352-d2404e064986-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m\" (UID: \"3b383bb9-0516-44ca-9352-d2404e064986\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:08:53.343438 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.343351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:08:53.476498 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.476384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"]
Jul 17 22:08:53.478770 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:08:53.478735    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b383bb9_0516_44ca_9352_d2404e064986.slice/crio-cd0124dba3be10f41d9cde710f1610a8d9fed3578ba0e0677099123133f8f467 WatchSource:0}: Error finding container cd0124dba3be10f41d9cde710f1610a8d9fed3578ba0e0677099123133f8f467: Status 404 returned error can't find the container with id cd0124dba3be10f41d9cde710f1610a8d9fed3578ba0e0677099123133f8f467
Jul 17 22:08:53.767623 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:53.767579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" event={"ID":"3b383bb9-0516-44ca-9352-d2404e064986","Type":"ContainerStarted","Data":"cd0124dba3be10f41d9cde710f1610a8d9fed3578ba0e0677099123133f8f467"}
Jul 17 22:08:54.774538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:54.774499    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b383bb9-0516-44ca-9352-d2404e064986" containerID="7d96f3ac6dd990cba65fa1a8fea6efaa3173d072271352c494de9353d4760499" exitCode=6
Jul 17 22:08:54.775026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:54.774557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" event={"ID":"3b383bb9-0516-44ca-9352-d2404e064986","Type":"ContainerDied","Data":"7d96f3ac6dd990cba65fa1a8fea6efaa3173d072271352c494de9353d4760499"}
Jul 17 22:08:54.775026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:54.774809    2563 scope.go:117] "RemoveContainer" containerID="7d96f3ac6dd990cba65fa1a8fea6efaa3173d072271352c494de9353d4760499"
Jul 17 22:08:55.780482 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:55.780449    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b383bb9-0516-44ca-9352-d2404e064986" containerID="05d496d3bdf6ba7ea7e5cd16b140d3d8b5fe12f5605b35c1edfb6b4389e73361" exitCode=6
Jul 17 22:08:55.780949 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:55.780536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" event={"ID":"3b383bb9-0516-44ca-9352-d2404e064986","Type":"ContainerDied","Data":"05d496d3bdf6ba7ea7e5cd16b140d3d8b5fe12f5605b35c1edfb6b4389e73361"}
Jul 17 22:08:55.780949 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:55.780629    2563 scope.go:117] "RemoveContainer" containerID="7d96f3ac6dd990cba65fa1a8fea6efaa3173d072271352c494de9353d4760499"
Jul 17 22:08:55.780949 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:55.780756    2563 scope.go:117] "RemoveContainer" containerID="05d496d3bdf6ba7ea7e5cd16b140d3d8b5fe12f5605b35c1edfb6b4389e73361"
Jul 17 22:08:55.781089 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:55.780984    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m_odh-ai-gateway-infra(3b383bb9-0516-44ca-9352-d2404e064986)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" podUID="3b383bb9-0516-44ca-9352-d2404e064986"
Jul 17 22:08:56.787148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:08:56.787119    2563 scope.go:117] "RemoveContainer" containerID="05d496d3bdf6ba7ea7e5cd16b140d3d8b5fe12f5605b35c1edfb6b4389e73361"
Jul 17 22:08:56.787548 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:08:56.787302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m_odh-ai-gateway-infra(3b383bb9-0516-44ca-9352-d2404e064986)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" podUID="3b383bb9-0516-44ca-9352-d2404e064986"
Jul 17 22:09:10.597077 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:10.596984    2563 scope.go:117] "RemoveContainer" containerID="05d496d3bdf6ba7ea7e5cd16b140d3d8b5fe12f5605b35c1edfb6b4389e73361"
Jul 17 22:09:11.861647 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:11.861608    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b383bb9-0516-44ca-9352-d2404e064986" containerID="e195acf75149ced09b5dc84e149ba9cb07faa69412be0d80727a9638641c7d0a" exitCode=6
Jul 17 22:09:11.861647 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:11.861625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" event={"ID":"3b383bb9-0516-44ca-9352-d2404e064986","Type":"ContainerDied","Data":"e195acf75149ced09b5dc84e149ba9cb07faa69412be0d80727a9638641c7d0a"}
Jul 17 22:09:11.862163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:11.861683    2563 scope.go:117] "RemoveContainer" containerID="05d496d3bdf6ba7ea7e5cd16b140d3d8b5fe12f5605b35c1edfb6b4389e73361"
Jul 17 22:09:11.862163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:11.862003    2563 scope.go:117] "RemoveContainer" containerID="e195acf75149ced09b5dc84e149ba9cb07faa69412be0d80727a9638641c7d0a"
Jul 17 22:09:11.862251 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:11.862208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m_odh-ai-gateway-infra(3b383bb9-0516-44ca-9352-d2404e064986)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" podUID="3b383bb9-0516-44ca-9352-d2404e064986"
Jul 17 22:09:12.885415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:12.885382    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"]
Jul 17 22:09:13.053426 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.053402    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:09:13.185339 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.185306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b383bb9-0516-44ca-9352-d2404e064986-maas-api-service-ca\") pod \"3b383bb9-0516-44ca-9352-d2404e064986\" (UID: \"3b383bb9-0516-44ca-9352-d2404e064986\") "
Jul 17 22:09:13.185724 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.185699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b383bb9-0516-44ca-9352-d2404e064986-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b383bb9-0516-44ca-9352-d2404e064986" (UID: "3b383bb9-0516-44ca-9352-d2404e064986"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:13.286705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.286664    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b383bb9-0516-44ca-9352-d2404e064986-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:09:13.874254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.874222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"
Jul 17 22:09:13.874254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.874228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m" event={"ID":"3b383bb9-0516-44ca-9352-d2404e064986","Type":"ContainerDied","Data":"cd0124dba3be10f41d9cde710f1610a8d9fed3578ba0e0677099123133f8f467"}
Jul 17 22:09:13.874475 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.874284    2563 scope.go:117] "RemoveContainer" containerID="e195acf75149ced09b5dc84e149ba9cb07faa69412be0d80727a9638641c7d0a"
Jul 17 22:09:13.898113 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.898079    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"]
Jul 17 22:09:13.901790 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:13.901763    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-d2q9m"]
Jul 17 22:09:14.603193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:14.603163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b383bb9-0516-44ca-9352-d2404e064986" path="/var/lib/kubelet/pods/3b383bb9-0516-44ca-9352-d2404e064986/volumes"
Jul 17 22:09:25.730024 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.729986    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"]
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730371    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730386    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730402    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730409    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730418    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730424    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730506    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98ec0e4-0596-4d8b-863c-7ee2202156ed" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730515    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:25.730630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.730524    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:25.733655 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.733629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:25.736007 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.735986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:09:25.740712 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.740679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"]
Jul 17 22:09:25.894483 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.894450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/471ffa41-95f6-4321-b55f-605d349f71db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8962d973-7bk9w\" (UID: \"471ffa41-95f6-4321-b55f-605d349f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:25.995698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.995590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/471ffa41-95f6-4321-b55f-605d349f71db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8962d973-7bk9w\" (UID: \"471ffa41-95f6-4321-b55f-605d349f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:25.996194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:25.996174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/471ffa41-95f6-4321-b55f-605d349f71db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8962d973-7bk9w\" (UID: \"471ffa41-95f6-4321-b55f-605d349f71db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:26.046260 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:26.046219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:26.183043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:26.183011    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"]
Jul 17 22:09:26.185401 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:09:26.185366    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471ffa41_95f6_4321_b55f_605d349f71db.slice/crio-98c07c2faef21c2f3616f6f3d54837bdbd35f38862c9371f4e71ff22db67225b WatchSource:0}: Error finding container 98c07c2faef21c2f3616f6f3d54837bdbd35f38862c9371f4e71ff22db67225b: Status 404 returned error can't find the container with id 98c07c2faef21c2f3616f6f3d54837bdbd35f38862c9371f4e71ff22db67225b
Jul 17 22:09:26.937983 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:26.937951    2563 generic.go:356] "Generic (PLEG): container finished" podID="471ffa41-95f6-4321-b55f-605d349f71db" containerID="b11a48cc9125d11bf15ee0cf80ed122b720c7032bed6e1911acb35ad8f5ebc23" exitCode=0
Jul 17 22:09:26.938341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:26.937992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w" event={"ID":"471ffa41-95f6-4321-b55f-605d349f71db","Type":"ContainerDied","Data":"b11a48cc9125d11bf15ee0cf80ed122b720c7032bed6e1911acb35ad8f5ebc23"}
Jul 17 22:09:26.938341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:26.938049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w" event={"ID":"471ffa41-95f6-4321-b55f-605d349f71db","Type":"ContainerStarted","Data":"98c07c2faef21c2f3616f6f3d54837bdbd35f38862c9371f4e71ff22db67225b"}
Jul 17 22:09:27.707569 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.707534    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"]
Jul 17 22:09:27.707929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.707917    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:27.707976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.707931    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:27.708022 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.708013    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b383bb9-0516-44ca-9352-d2404e064986" containerName="revoke-keys"
Jul 17 22:09:27.710991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.710974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:27.717527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.717503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"]
Jul 17 22:09:27.810051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.810009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5677d00d-dbb5-45c7-a8a6-c61e389d5e94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh\" (UID: \"5677d00d-dbb5-45c7-a8a6-c61e389d5e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:27.911094 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.911052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5677d00d-dbb5-45c7-a8a6-c61e389d5e94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh\" (UID: \"5677d00d-dbb5-45c7-a8a6-c61e389d5e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:27.911714 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:27.911695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5677d00d-dbb5-45c7-a8a6-c61e389d5e94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh\" (UID: \"5677d00d-dbb5-45c7-a8a6-c61e389d5e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:28.023174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.023145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:28.102019 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.101991    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:28.163562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.163528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"]
Jul 17 22:09:28.166440 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:09:28.166405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5677d00d_dbb5_45c7_a8a6_c61e389d5e94.slice/crio-96dacd3c979faf05fed63ec4e8442a4511127a82b06e6bcb3156d9e187fdea52 WatchSource:0}: Error finding container 96dacd3c979faf05fed63ec4e8442a4511127a82b06e6bcb3156d9e187fdea52: Status 404 returned error can't find the container with id 96dacd3c979faf05fed63ec4e8442a4511127a82b06e6bcb3156d9e187fdea52
Jul 17 22:09:28.214111 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.214082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/471ffa41-95f6-4321-b55f-605d349f71db-maas-api-service-ca\") pod \"471ffa41-95f6-4321-b55f-605d349f71db\" (UID: \"471ffa41-95f6-4321-b55f-605d349f71db\") "
Jul 17 22:09:28.214432 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.214409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471ffa41-95f6-4321-b55f-605d349f71db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "471ffa41-95f6-4321-b55f-605d349f71db" (UID: "471ffa41-95f6-4321-b55f-605d349f71db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:28.315701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.315590    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/471ffa41-95f6-4321-b55f-605d349f71db-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:09:28.949271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.949239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w" event={"ID":"471ffa41-95f6-4321-b55f-605d349f71db","Type":"ContainerDied","Data":"98c07c2faef21c2f3616f6f3d54837bdbd35f38862c9371f4e71ff22db67225b"}
Jul 17 22:09:28.949451 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.949293    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c07c2faef21c2f3616f6f3d54837bdbd35f38862c9371f4e71ff22db67225b"
Jul 17 22:09:28.949451 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.949248    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8962d973-7bk9w"
Jul 17 22:09:28.950808 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.950780    2563 generic.go:356] "Generic (PLEG): container finished" podID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerID="c4b91ff6550e4651ee9c776ea840ba4e78e46d061da30ef6015855c2f14c3b35" exitCode=6
Jul 17 22:09:28.950972 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.950830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" event={"ID":"5677d00d-dbb5-45c7-a8a6-c61e389d5e94","Type":"ContainerDied","Data":"c4b91ff6550e4651ee9c776ea840ba4e78e46d061da30ef6015855c2f14c3b35"}
Jul 17 22:09:28.950972 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.950874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" event={"ID":"5677d00d-dbb5-45c7-a8a6-c61e389d5e94","Type":"ContainerStarted","Data":"96dacd3c979faf05fed63ec4e8442a4511127a82b06e6bcb3156d9e187fdea52"}
Jul 17 22:09:28.951114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:28.951016    2563 scope.go:117] "RemoveContainer" containerID="c4b91ff6550e4651ee9c776ea840ba4e78e46d061da30ef6015855c2f14c3b35"
Jul 17 22:09:29.957080 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:29.957046    2563 generic.go:356] "Generic (PLEG): container finished" podID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerID="f145f4d619b80fc7e8ba5ce6705562566bc0c666b0286ccdc5023b4651db044f" exitCode=6
Jul 17 22:09:29.957489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:29.957133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" event={"ID":"5677d00d-dbb5-45c7-a8a6-c61e389d5e94","Type":"ContainerDied","Data":"f145f4d619b80fc7e8ba5ce6705562566bc0c666b0286ccdc5023b4651db044f"}
Jul 17 22:09:29.957489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:29.957191    2563 scope.go:117] "RemoveContainer" containerID="c4b91ff6550e4651ee9c776ea840ba4e78e46d061da30ef6015855c2f14c3b35"
Jul 17 22:09:29.957489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:29.957408    2563 scope.go:117] "RemoveContainer" containerID="f145f4d619b80fc7e8ba5ce6705562566bc0c666b0286ccdc5023b4651db044f"
Jul 17 22:09:29.957641 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:29.957626    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh_odh-ai-gateway-infra(5677d00d-dbb5-45c7-a8a6-c61e389d5e94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94"
Jul 17 22:09:30.963558 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:30.963529    2563 scope.go:117] "RemoveContainer" containerID="f145f4d619b80fc7e8ba5ce6705562566bc0c666b0286ccdc5023b4651db044f"
Jul 17 22:09:30.963967 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:30.963768    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh_odh-ai-gateway-infra(5677d00d-dbb5-45c7-a8a6-c61e389d5e94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94"
Jul 17 22:09:42.597393 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:42.597364    2563 scope.go:117] "RemoveContainer" containerID="f145f4d619b80fc7e8ba5ce6705562566bc0c666b0286ccdc5023b4651db044f"
Jul 17 22:09:43.020330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:43.020295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" event={"ID":"5677d00d-dbb5-45c7-a8a6-c61e389d5e94","Type":"ContainerStarted","Data":"c45623a5e91f20f18a84005c2866f7fef6afc89dcebb562d182a9119d4dad9ef"}
Jul 17 22:09:43.037123 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:43.037072    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" podStartSLOduration=16.037057343 podStartE2EDuration="16.037057343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:43.034821122 +0000 UTC m=+2546.947061450" watchObservedRunningTime="2026-07-17 22:09:43.037057343 +0000 UTC m=+2546.949297660"
Jul 17 22:09:44.026445 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:44.026404    2563 generic.go:356] "Generic (PLEG): container finished" podID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerID="c45623a5e91f20f18a84005c2866f7fef6afc89dcebb562d182a9119d4dad9ef" exitCode=6
Jul 17 22:09:44.026947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:44.026475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" event={"ID":"5677d00d-dbb5-45c7-a8a6-c61e389d5e94","Type":"ContainerDied","Data":"c45623a5e91f20f18a84005c2866f7fef6afc89dcebb562d182a9119d4dad9ef"}
Jul 17 22:09:44.026947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:44.026558    2563 scope.go:117] "RemoveContainer" containerID="f145f4d619b80fc7e8ba5ce6705562566bc0c666b0286ccdc5023b4651db044f"
Jul 17 22:09:44.026947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:44.026864    2563 scope.go:117] "RemoveContainer" containerID="c45623a5e91f20f18a84005c2866f7fef6afc89dcebb562d182a9119d4dad9ef"
Jul 17 22:09:44.027135 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:44.027113    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh_odh-ai-gateway-infra(5677d00d-dbb5-45c7-a8a6-c61e389d5e94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94"
Jul 17 22:09:44.051336 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:44.051311    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"]
Jul 17 22:09:45.171555 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.171530    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:45.257671 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.257639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5677d00d-dbb5-45c7-a8a6-c61e389d5e94-maas-api-service-ca\") pod \"5677d00d-dbb5-45c7-a8a6-c61e389d5e94\" (UID: \"5677d00d-dbb5-45c7-a8a6-c61e389d5e94\") "
Jul 17 22:09:45.257985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.257963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5677d00d-dbb5-45c7-a8a6-c61e389d5e94-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5677d00d-dbb5-45c7-a8a6-c61e389d5e94" (UID: "5677d00d-dbb5-45c7-a8a6-c61e389d5e94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:45.358847 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.358775    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5677d00d-dbb5-45c7-a8a6-c61e389d5e94-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:09:45.625096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"]
Jul 17 22:09:45.625369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:45.625419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:45.625419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625385    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471ffa41-95f6-4321-b55f-605d349f71db" containerName="revoke-keys"
Jul 17 22:09:45.625419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625391    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="471ffa41-95f6-4321-b55f-605d349f71db" containerName="revoke-keys"
Jul 17 22:09:45.625419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625407    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:45.625419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625414    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:45.625625 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625488    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:45.625625 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625497    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="471ffa41-95f6-4321-b55f-605d349f71db" containerName="revoke-keys"
Jul 17 22:09:45.625625 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.625503    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:45.628576 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.628559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:09:45.636466 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.636438    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"]
Jul 17 22:09:45.661797 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.661761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d640f690-abf4-42d0-9533-7fb426fa78e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj\" (UID: \"d640f690-abf4-42d0-9533-7fb426fa78e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:09:45.763126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.763094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d640f690-abf4-42d0-9533-7fb426fa78e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj\" (UID: \"d640f690-abf4-42d0-9533-7fb426fa78e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:09:45.763757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.763737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d640f690-abf4-42d0-9533-7fb426fa78e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj\" (UID: \"d640f690-abf4-42d0-9533-7fb426fa78e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:09:45.940050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:45.940018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:09:46.039162 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.039128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh" event={"ID":"5677d00d-dbb5-45c7-a8a6-c61e389d5e94","Type":"ContainerDied","Data":"96dacd3c979faf05fed63ec4e8442a4511127a82b06e6bcb3156d9e187fdea52"}
Jul 17 22:09:46.039333 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.039196    2563 scope.go:117] "RemoveContainer" containerID="c45623a5e91f20f18a84005c2866f7fef6afc89dcebb562d182a9119d4dad9ef"
Jul 17 22:09:46.039333 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.039145    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"
Jul 17 22:09:46.065474 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.065447    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"]
Jul 17 22:09:46.071584 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.071559    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-jrgmh"]
Jul 17 22:09:46.079102 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.079080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"]
Jul 17 22:09:46.082046 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:09:46.082021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd640f690_abf4_42d0_9533_7fb426fa78e3.slice/crio-eb1a25dfffdc0ce3381b00257001347a4c17508317706e20fa0320a67cc36192 WatchSource:0}: Error finding container eb1a25dfffdc0ce3381b00257001347a4c17508317706e20fa0320a67cc36192: Status 404 returned error can't find the container with id eb1a25dfffdc0ce3381b00257001347a4c17508317706e20fa0320a67cc36192
Jul 17 22:09:46.604357 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:46.604277    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" path="/var/lib/kubelet/pods/5677d00d-dbb5-45c7-a8a6-c61e389d5e94/volumes"
Jul 17 22:09:47.045658 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:47.045627    2563 generic.go:356] "Generic (PLEG): container finished" podID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerID="9e3762fef223b47ddab0e2743d4e9583b9a392dfc4b6b79564a89726a2d5996b" exitCode=6
Jul 17 22:09:47.045833 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:47.045714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" event={"ID":"d640f690-abf4-42d0-9533-7fb426fa78e3","Type":"ContainerDied","Data":"9e3762fef223b47ddab0e2743d4e9583b9a392dfc4b6b79564a89726a2d5996b"}
Jul 17 22:09:47.045833 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:47.045766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" event={"ID":"d640f690-abf4-42d0-9533-7fb426fa78e3","Type":"ContainerStarted","Data":"eb1a25dfffdc0ce3381b00257001347a4c17508317706e20fa0320a67cc36192"}
Jul 17 22:09:47.045929 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:47.045888    2563 scope.go:117] "RemoveContainer" containerID="9e3762fef223b47ddab0e2743d4e9583b9a392dfc4b6b79564a89726a2d5996b"
Jul 17 22:09:48.053538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:48.053451    2563 generic.go:356] "Generic (PLEG): container finished" podID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerID="1e1072eba1c1342e395bc26f98db365a56bfb5459ea6a7dd3bd6f77461daa395" exitCode=6
Jul 17 22:09:48.053538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:48.053527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" event={"ID":"d640f690-abf4-42d0-9533-7fb426fa78e3","Type":"ContainerDied","Data":"1e1072eba1c1342e395bc26f98db365a56bfb5459ea6a7dd3bd6f77461daa395"}
Jul 17 22:09:48.054026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:48.053643    2563 scope.go:117] "RemoveContainer" containerID="9e3762fef223b47ddab0e2743d4e9583b9a392dfc4b6b79564a89726a2d5996b"
Jul 17 22:09:48.054026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:48.053844    2563 scope.go:117] "RemoveContainer" containerID="1e1072eba1c1342e395bc26f98db365a56bfb5459ea6a7dd3bd6f77461daa395"
Jul 17 22:09:48.054107 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:48.054042    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj_odh-ai-gateway-infra(d640f690-abf4-42d0-9533-7fb426fa78e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3"
Jul 17 22:09:49.059918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:49.059893    2563 scope.go:117] "RemoveContainer" containerID="1e1072eba1c1342e395bc26f98db365a56bfb5459ea6a7dd3bd6f77461daa395"
Jul 17 22:09:49.060314 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:49.060070    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj_odh-ai-gateway-infra(d640f690-abf4-42d0-9533-7fb426fa78e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3"
Jul 17 22:09:53.084180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.084146    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"]
Jul 17 22:09:53.084578 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.084519    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:53.084578 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.084530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:53.084686 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.084652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5677d00d-dbb5-45c7-a8a6-c61e389d5e94" containerName="revoke-keys"
Jul 17 22:09:53.087566 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.087550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:09:53.094535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.094509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"]
Jul 17 22:09:53.125736 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.125699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19d6759-b49e-40de-8472-5f78630aac0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52\" (UID: \"f19d6759-b49e-40de-8472-5f78630aac0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:09:53.226822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.226781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19d6759-b49e-40de-8472-5f78630aac0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52\" (UID: \"f19d6759-b49e-40de-8472-5f78630aac0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:09:53.227444 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.227423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19d6759-b49e-40de-8472-5f78630aac0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52\" (UID: \"f19d6759-b49e-40de-8472-5f78630aac0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:09:53.399462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.399377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:09:53.539121 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:53.539098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"]
Jul 17 22:09:53.541122 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:09:53.541096    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19d6759_b49e_40de_8472_5f78630aac0a.slice/crio-94c4286ecaf586768f2b948b269153eb8706393f6b6a4124d623e911e5575d8a WatchSource:0}: Error finding container 94c4286ecaf586768f2b948b269153eb8706393f6b6a4124d623e911e5575d8a: Status 404 returned error can't find the container with id 94c4286ecaf586768f2b948b269153eb8706393f6b6a4124d623e911e5575d8a
Jul 17 22:09:54.084057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:54.083966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" event={"ID":"f19d6759-b49e-40de-8472-5f78630aac0a","Type":"ContainerStarted","Data":"46f4c681216e48988c4445db9e2a230e74070ad195944f8415290a006b6454eb"}
Jul 17 22:09:54.084057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:54.084022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" event={"ID":"f19d6759-b49e-40de-8472-5f78630aac0a","Type":"ContainerStarted","Data":"94c4286ecaf586768f2b948b269153eb8706393f6b6a4124d623e911e5575d8a"}
Jul 17 22:09:54.102211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:54.102157    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" podStartSLOduration=1.102141153 podStartE2EDuration="1.102141153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:54.100716508 +0000 UTC m=+2558.012956809" watchObservedRunningTime="2026-07-17 22:09:54.102141153 +0000 UTC m=+2558.014381442"
Jul 17 22:09:55.090205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:55.090171    2563 generic.go:356] "Generic (PLEG): container finished" podID="f19d6759-b49e-40de-8472-5f78630aac0a" containerID="46f4c681216e48988c4445db9e2a230e74070ad195944f8415290a006b6454eb" exitCode=6
Jul 17 22:09:55.090734 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:55.090257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" event={"ID":"f19d6759-b49e-40de-8472-5f78630aac0a","Type":"ContainerDied","Data":"46f4c681216e48988c4445db9e2a230e74070ad195944f8415290a006b6454eb"}
Jul 17 22:09:55.090734 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:55.090467    2563 scope.go:117] "RemoveContainer" containerID="46f4c681216e48988c4445db9e2a230e74070ad195944f8415290a006b6454eb"
Jul 17 22:09:56.097479 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:56.097448    2563 generic.go:356] "Generic (PLEG): container finished" podID="f19d6759-b49e-40de-8472-5f78630aac0a" containerID="0fba10c5a2e478481de867551e663c6c1cb8191599f4d6de26386ea23841436e" exitCode=6
Jul 17 22:09:56.097981 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:56.097530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" event={"ID":"f19d6759-b49e-40de-8472-5f78630aac0a","Type":"ContainerDied","Data":"0fba10c5a2e478481de867551e663c6c1cb8191599f4d6de26386ea23841436e"}
Jul 17 22:09:56.097981 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:56.097623    2563 scope.go:117] "RemoveContainer" containerID="46f4c681216e48988c4445db9e2a230e74070ad195944f8415290a006b6454eb"
Jul 17 22:09:56.097981 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:56.097807    2563 scope.go:117] "RemoveContainer" containerID="0fba10c5a2e478481de867551e663c6c1cb8191599f4d6de26386ea23841436e"
Jul 17 22:09:56.098129 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:56.098039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52_odh-ai-gateway-infra(f19d6759-b49e-40de-8472-5f78630aac0a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" podUID="f19d6759-b49e-40de-8472-5f78630aac0a"
Jul 17 22:09:57.104304 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:09:57.104272    2563 scope.go:117] "RemoveContainer" containerID="0fba10c5a2e478481de867551e663c6c1cb8191599f4d6de26386ea23841436e"
Jul 17 22:09:57.104790 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:09:57.104494    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52_odh-ai-gateway-infra(f19d6759-b49e-40de-8472-5f78630aac0a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" podUID="f19d6759-b49e-40de-8472-5f78630aac0a"
Jul 17 22:10:02.596799 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:02.596769    2563 scope.go:117] "RemoveContainer" containerID="1e1072eba1c1342e395bc26f98db365a56bfb5459ea6a7dd3bd6f77461daa395"
Jul 17 22:10:03.136247 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:03.136161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" event={"ID":"d640f690-abf4-42d0-9533-7fb426fa78e3","Type":"ContainerStarted","Data":"6a498657f01fe8661af2785ea852e199bc45db7153c3ffcb85f7192ce18324c9"}
Jul 17 22:10:03.150720 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:03.150677    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" podStartSLOduration=18.150658822 podStartE2EDuration="18.150658822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:03.150008771 +0000 UTC m=+2567.062249073" watchObservedRunningTime="2026-07-17 22:10:03.150658822 +0000 UTC m=+2567.062899111"
Jul 17 22:10:04.142866 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:04.142835    2563 generic.go:356] "Generic (PLEG): container finished" podID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerID="6a498657f01fe8661af2785ea852e199bc45db7153c3ffcb85f7192ce18324c9" exitCode=6
Jul 17 22:10:04.143297 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:04.142883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" event={"ID":"d640f690-abf4-42d0-9533-7fb426fa78e3","Type":"ContainerDied","Data":"6a498657f01fe8661af2785ea852e199bc45db7153c3ffcb85f7192ce18324c9"}
Jul 17 22:10:04.143297 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:04.142942    2563 scope.go:117] "RemoveContainer" containerID="1e1072eba1c1342e395bc26f98db365a56bfb5459ea6a7dd3bd6f77461daa395"
Jul 17 22:10:04.143297 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:04.143214    2563 scope.go:117] "RemoveContainer" containerID="6a498657f01fe8661af2785ea852e199bc45db7153c3ffcb85f7192ce18324c9"
Jul 17 22:10:04.143501 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:04.143405    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj_odh-ai-gateway-infra(d640f690-abf4-42d0-9533-7fb426fa78e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3"
Jul 17 22:10:04.187102 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:04.187068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"]
Jul 17 22:10:05.289428 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:05.289403    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:10:05.433138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:05.433112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d640f690-abf4-42d0-9533-7fb426fa78e3-maas-api-service-ca\") pod \"d640f690-abf4-42d0-9533-7fb426fa78e3\" (UID: \"d640f690-abf4-42d0-9533-7fb426fa78e3\") "
Jul 17 22:10:05.433422 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:05.433399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d640f690-abf4-42d0-9533-7fb426fa78e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d640f690-abf4-42d0-9533-7fb426fa78e3" (UID: "d640f690-abf4-42d0-9533-7fb426fa78e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:05.533777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:05.533744    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d640f690-abf4-42d0-9533-7fb426fa78e3-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:10:06.154411 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:06.154374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj" event={"ID":"d640f690-abf4-42d0-9533-7fb426fa78e3","Type":"ContainerDied","Data":"eb1a25dfffdc0ce3381b00257001347a4c17508317706e20fa0320a67cc36192"}
Jul 17 22:10:06.154411 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:06.154392    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"
Jul 17 22:10:06.154688 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:06.154434    2563 scope.go:117] "RemoveContainer" containerID="6a498657f01fe8661af2785ea852e199bc45db7153c3ffcb85f7192ce18324c9"
Jul 17 22:10:06.175209 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:06.175173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"]
Jul 17 22:10:06.179666 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:06.179643    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-hszjj"]
Jul 17 22:10:06.603067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:06.603037    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" path="/var/lib/kubelet/pods/d640f690-abf4-42d0-9533-7fb426fa78e3/volumes"
Jul 17 22:10:10.596949 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:10.596915    2563 scope.go:117] "RemoveContainer" containerID="0fba10c5a2e478481de867551e663c6c1cb8191599f4d6de26386ea23841436e"
Jul 17 22:10:11.181518 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:11.181492    2563 generic.go:356] "Generic (PLEG): container finished" podID="f19d6759-b49e-40de-8472-5f78630aac0a" containerID="504b454774d22e735443e61b2bad586937e338bed6d533c67ed794c7ff3ad002" exitCode=6
Jul 17 22:10:11.181673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:11.181525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" event={"ID":"f19d6759-b49e-40de-8472-5f78630aac0a","Type":"ContainerDied","Data":"504b454774d22e735443e61b2bad586937e338bed6d533c67ed794c7ff3ad002"}
Jul 17 22:10:11.181673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:11.181570    2563 scope.go:117] "RemoveContainer" containerID="0fba10c5a2e478481de867551e663c6c1cb8191599f4d6de26386ea23841436e"
Jul 17 22:10:11.181863 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:11.181847    2563 scope.go:117] "RemoveContainer" containerID="504b454774d22e735443e61b2bad586937e338bed6d533c67ed794c7ff3ad002"
Jul 17 22:10:11.182079 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:11.182057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52_odh-ai-gateway-infra(f19d6759-b49e-40de-8472-5f78630aac0a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" podUID="f19d6759-b49e-40de-8472-5f78630aac0a"
Jul 17 22:10:12.208543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:12.208513    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"]
Jul 17 22:10:12.344803 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:12.344779    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:10:12.493324 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:12.493241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19d6759-b49e-40de-8472-5f78630aac0a-maas-api-service-ca\") pod \"f19d6759-b49e-40de-8472-5f78630aac0a\" (UID: \"f19d6759-b49e-40de-8472-5f78630aac0a\") "
Jul 17 22:10:12.493618 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:12.493579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19d6759-b49e-40de-8472-5f78630aac0a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f19d6759-b49e-40de-8472-5f78630aac0a" (UID: "f19d6759-b49e-40de-8472-5f78630aac0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:12.594693 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:12.594660    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19d6759-b49e-40de-8472-5f78630aac0a-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:10:13.193756 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:13.193726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"
Jul 17 22:10:13.193968 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:13.193718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52" event={"ID":"f19d6759-b49e-40de-8472-5f78630aac0a","Type":"ContainerDied","Data":"94c4286ecaf586768f2b948b269153eb8706393f6b6a4124d623e911e5575d8a"}
Jul 17 22:10:13.193968 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:13.193854    2563 scope.go:117] "RemoveContainer" containerID="504b454774d22e735443e61b2bad586937e338bed6d533c67ed794c7ff3ad002"
Jul 17 22:10:13.209303 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:13.209278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"]
Jul 17 22:10:13.213298 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:13.213277    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-bbx52"]
Jul 17 22:10:14.602815 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:14.602785    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" path="/var/lib/kubelet/pods/f19d6759-b49e-40de-8472-5f78630aac0a/volumes"
Jul 17 22:10:17.750454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"]
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750814    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750825    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750853    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750858    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750865    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.750873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750869    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.751140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750886    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:17.751140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:17.751140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750945    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.751140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750952    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:17.751140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750958    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:17.751140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.750963    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:17.755026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.755007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:17.757277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.757257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:10:17.760277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.760248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"]
Jul 17 22:10:17.838735 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.838700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc9c72-aed0-43c9-860b-55128b3f8eb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw\" (UID: \"25bc9c72-aed0-43c9-860b-55128b3f8eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:17.939812 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.939774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc9c72-aed0-43c9-860b-55128b3f8eb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw\" (UID: \"25bc9c72-aed0-43c9-860b-55128b3f8eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:17.940369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:17.940350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc9c72-aed0-43c9-860b-55128b3f8eb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw\" (UID: \"25bc9c72-aed0-43c9-860b-55128b3f8eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:18.067287 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:18.067204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:18.200522 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:18.200495    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"]
Jul 17 22:10:18.204010 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:10:18.203973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bc9c72_aed0_43c9_860b_55128b3f8eb4.slice/crio-5bdc4490877457ee770f7abbe3cb77123bbf83d00989daf716338fc4f14a44b1 WatchSource:0}: Error finding container 5bdc4490877457ee770f7abbe3cb77123bbf83d00989daf716338fc4f14a44b1: Status 404 returned error can't find the container with id 5bdc4490877457ee770f7abbe3cb77123bbf83d00989daf716338fc4f14a44b1
Jul 17 22:10:18.221389 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:18.221361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" event={"ID":"25bc9c72-aed0-43c9-860b-55128b3f8eb4","Type":"ContainerStarted","Data":"5bdc4490877457ee770f7abbe3cb77123bbf83d00989daf716338fc4f14a44b1"}
Jul 17 22:10:19.228191 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:19.228160    2563 generic.go:356] "Generic (PLEG): container finished" podID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerID="86f98949e7ec44b27f73fdd6ae8757f3c79d8f96f91342592b531fccd64ccde5" exitCode=6
Jul 17 22:10:19.228586 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:19.228264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" event={"ID":"25bc9c72-aed0-43c9-860b-55128b3f8eb4","Type":"ContainerDied","Data":"86f98949e7ec44b27f73fdd6ae8757f3c79d8f96f91342592b531fccd64ccde5"}
Jul 17 22:10:19.228586 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:19.228386    2563 scope.go:117] "RemoveContainer" containerID="86f98949e7ec44b27f73fdd6ae8757f3c79d8f96f91342592b531fccd64ccde5"
Jul 17 22:10:20.234783 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:20.234747    2563 generic.go:356] "Generic (PLEG): container finished" podID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerID="10287ee6ee998592065697994b3de6396e420920250c9b9d25e1f484307f12c6" exitCode=6
Jul 17 22:10:20.235223 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:20.234836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" event={"ID":"25bc9c72-aed0-43c9-860b-55128b3f8eb4","Type":"ContainerDied","Data":"10287ee6ee998592065697994b3de6396e420920250c9b9d25e1f484307f12c6"}
Jul 17 22:10:20.235223 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:20.234905    2563 scope.go:117] "RemoveContainer" containerID="86f98949e7ec44b27f73fdd6ae8757f3c79d8f96f91342592b531fccd64ccde5"
Jul 17 22:10:20.235223 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:20.235145    2563 scope.go:117] "RemoveContainer" containerID="10287ee6ee998592065697994b3de6396e420920250c9b9d25e1f484307f12c6"
Jul 17 22:10:20.235371 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:20.235353    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw_odh-ai-gateway-infra(25bc9c72-aed0-43c9-860b-55128b3f8eb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4"
Jul 17 22:10:21.241771 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:21.241745    2563 scope.go:117] "RemoveContainer" containerID="10287ee6ee998592065697994b3de6396e420920250c9b9d25e1f484307f12c6"
Jul 17 22:10:21.242148 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:21.241944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw_odh-ai-gateway-infra(25bc9c72-aed0-43c9-860b-55128b3f8eb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4"
Jul 17 22:10:33.597233 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:33.597199    2563 scope.go:117] "RemoveContainer" containerID="10287ee6ee998592065697994b3de6396e420920250c9b9d25e1f484307f12c6"
Jul 17 22:10:34.304097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:34.304066    2563 generic.go:356] "Generic (PLEG): container finished" podID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerID="bdb9678e8a30c8e749bed5c7e65f5df64ec3c3fea03e13cbb1410ceed0ca2fff" exitCode=6
Jul 17 22:10:34.304296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:34.304135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" event={"ID":"25bc9c72-aed0-43c9-860b-55128b3f8eb4","Type":"ContainerDied","Data":"bdb9678e8a30c8e749bed5c7e65f5df64ec3c3fea03e13cbb1410ceed0ca2fff"}
Jul 17 22:10:34.304296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:34.304196    2563 scope.go:117] "RemoveContainer" containerID="10287ee6ee998592065697994b3de6396e420920250c9b9d25e1f484307f12c6"
Jul 17 22:10:34.304560 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:34.304542    2563 scope.go:117] "RemoveContainer" containerID="bdb9678e8a30c8e749bed5c7e65f5df64ec3c3fea03e13cbb1410ceed0ca2fff"
Jul 17 22:10:34.304803 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:34.304764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw_odh-ai-gateway-infra(25bc9c72-aed0-43c9-860b-55128b3f8eb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4"
Jul 17 22:10:35.330968 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:35.330935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"]
Jul 17 22:10:35.469141 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:35.469120    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:35.582626 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:35.582510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc9c72-aed0-43c9-860b-55128b3f8eb4-maas-api-service-ca\") pod \"25bc9c72-aed0-43c9-860b-55128b3f8eb4\" (UID: \"25bc9c72-aed0-43c9-860b-55128b3f8eb4\") "
Jul 17 22:10:35.582902 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:35.582876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bc9c72-aed0-43c9-860b-55128b3f8eb4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25bc9c72-aed0-43c9-860b-55128b3f8eb4" (UID: "25bc9c72-aed0-43c9-860b-55128b3f8eb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:35.683163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:35.683133    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25bc9c72-aed0-43c9-860b-55128b3f8eb4-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:10:36.317529 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:36.317491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw" event={"ID":"25bc9c72-aed0-43c9-860b-55128b3f8eb4","Type":"ContainerDied","Data":"5bdc4490877457ee770f7abbe3cb77123bbf83d00989daf716338fc4f14a44b1"}
Jul 17 22:10:36.317529 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:36.317517    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"
Jul 17 22:10:36.317813 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:36.317569    2563 scope.go:117] "RemoveContainer" containerID="bdb9678e8a30c8e749bed5c7e65f5df64ec3c3fea03e13cbb1410ceed0ca2fff"
Jul 17 22:10:36.337398 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:36.337367    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"]
Jul 17 22:10:36.341970 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:36.341948    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bpdvw"]
Jul 17 22:10:36.602960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:36.602879    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" path="/var/lib/kubelet/pods/25bc9c72-aed0-43c9-860b-55128b3f8eb4/volumes"
Jul 17 22:10:45.668395 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668303    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"]
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668860    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668907    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668916    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668928    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668961    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:45.669008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.668969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:45.669415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.669055    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.669069    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19d6759-b49e-40de-8472-5f78630aac0a" containerName="revoke-keys"
Jul 17 22:10:45.669415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.669080    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.669415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.669091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d640f690-abf4-42d0-9533-7fb426fa78e3" containerName="revoke-keys"
Jul 17 22:10:45.669415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.669102    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bc9c72-aed0-43c9-860b-55128b3f8eb4" containerName="revoke-keys"
Jul 17 22:10:45.673743 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.673720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:10:45.676050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.676027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:10:45.679407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.679384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"]
Jul 17 22:10:45.772644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.772612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6387c744-c265-44e6-924a-e779f9e299dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp\" (UID: \"6387c744-c265-44e6-924a-e779f9e299dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:10:45.873672 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.873637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6387c744-c265-44e6-924a-e779f9e299dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp\" (UID: \"6387c744-c265-44e6-924a-e779f9e299dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:10:45.874230 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.874208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6387c744-c265-44e6-924a-e779f9e299dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp\" (UID: \"6387c744-c265-44e6-924a-e779f9e299dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:10:45.986422 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:45.986344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:10:46.123105 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:46.123072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"]
Jul 17 22:10:46.126698 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:10:46.126656    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6387c744_c265_44e6_924a_e779f9e299dc.slice/crio-bd957f39507aedb38d85fbcc9e4a3c10389b736261a4ddaa0aaa2af80d515cc1 WatchSource:0}: Error finding container bd957f39507aedb38d85fbcc9e4a3c10389b736261a4ddaa0aaa2af80d515cc1: Status 404 returned error can't find the container with id bd957f39507aedb38d85fbcc9e4a3c10389b736261a4ddaa0aaa2af80d515cc1
Jul 17 22:10:46.366551 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:46.366478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" event={"ID":"6387c744-c265-44e6-924a-e779f9e299dc","Type":"ContainerStarted","Data":"bd957f39507aedb38d85fbcc9e4a3c10389b736261a4ddaa0aaa2af80d515cc1"}
Jul 17 22:10:47.372263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:47.372225    2563 generic.go:356] "Generic (PLEG): container finished" podID="6387c744-c265-44e6-924a-e779f9e299dc" containerID="cef303812fb38a892bd9681bd861e773ad9bb05bcdcc4c793230eacac523e238" exitCode=6
Jul 17 22:10:47.372668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:47.372297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" event={"ID":"6387c744-c265-44e6-924a-e779f9e299dc","Type":"ContainerDied","Data":"cef303812fb38a892bd9681bd861e773ad9bb05bcdcc4c793230eacac523e238"}
Jul 17 22:10:47.372668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:47.372482    2563 scope.go:117] "RemoveContainer" containerID="cef303812fb38a892bd9681bd861e773ad9bb05bcdcc4c793230eacac523e238"
Jul 17 22:10:48.378920 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:48.378830    2563 generic.go:356] "Generic (PLEG): container finished" podID="6387c744-c265-44e6-924a-e779f9e299dc" containerID="a7e85aa28bd04cb8416418b23e17a814a0b60e278539fb65e07a9993349a5de7" exitCode=6
Jul 17 22:10:48.378920 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:48.378882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" event={"ID":"6387c744-c265-44e6-924a-e779f9e299dc","Type":"ContainerDied","Data":"a7e85aa28bd04cb8416418b23e17a814a0b60e278539fb65e07a9993349a5de7"}
Jul 17 22:10:48.379390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:48.378943    2563 scope.go:117] "RemoveContainer" containerID="cef303812fb38a892bd9681bd861e773ad9bb05bcdcc4c793230eacac523e238"
Jul 17 22:10:48.379390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:48.379084    2563 scope.go:117] "RemoveContainer" containerID="a7e85aa28bd04cb8416418b23e17a814a0b60e278539fb65e07a9993349a5de7"
Jul 17 22:10:48.379390 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:48.379311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp_odh-ai-gateway-infra(6387c744-c265-44e6-924a-e779f9e299dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" podUID="6387c744-c265-44e6-924a-e779f9e299dc"
Jul 17 22:10:49.385044 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:49.385017    2563 scope.go:117] "RemoveContainer" containerID="a7e85aa28bd04cb8416418b23e17a814a0b60e278539fb65e07a9993349a5de7"
Jul 17 22:10:49.385435 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:49.385234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp_odh-ai-gateway-infra(6387c744-c265-44e6-924a-e779f9e299dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" podUID="6387c744-c265-44e6-924a-e779f9e299dc"
Jul 17 22:10:53.128783 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.128751    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"]
Jul 17 22:10:53.131947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.131929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:10:53.139002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.138970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"]
Jul 17 22:10:53.235023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.234984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a6addd-d5bd-4864-b8ff-95dd65f03e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2\" (UID: \"e9a6addd-d5bd-4864-b8ff-95dd65f03e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:10:53.335689 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.335650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a6addd-d5bd-4864-b8ff-95dd65f03e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2\" (UID: \"e9a6addd-d5bd-4864-b8ff-95dd65f03e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:10:53.336248 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.336230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a6addd-d5bd-4864-b8ff-95dd65f03e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2\" (UID: \"e9a6addd-d5bd-4864-b8ff-95dd65f03e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:10:53.444009 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.443984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:10:53.580447 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:53.580393    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"]
Jul 17 22:10:53.583456 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:10:53.583429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a6addd_d5bd_4864_b8ff_95dd65f03e16.slice/crio-a00ee3dac71bf22d1cb6089572650effb47bbc3500dcd065e5a1f34cb606dc19 WatchSource:0}: Error finding container a00ee3dac71bf22d1cb6089572650effb47bbc3500dcd065e5a1f34cb606dc19: Status 404 returned error can't find the container with id a00ee3dac71bf22d1cb6089572650effb47bbc3500dcd065e5a1f34cb606dc19
Jul 17 22:10:54.409398 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.409362    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerID="7223ee284c91905383e55f1f2e9eb2c2bbed625ccfaf5f399d0a3449dd7c25c1" exitCode=6
Jul 17 22:10:54.409811 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.409408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" event={"ID":"e9a6addd-d5bd-4864-b8ff-95dd65f03e16","Type":"ContainerDied","Data":"7223ee284c91905383e55f1f2e9eb2c2bbed625ccfaf5f399d0a3449dd7c25c1"}
Jul 17 22:10:54.409811 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.409466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" event={"ID":"e9a6addd-d5bd-4864-b8ff-95dd65f03e16","Type":"ContainerStarted","Data":"a00ee3dac71bf22d1cb6089572650effb47bbc3500dcd065e5a1f34cb606dc19"}
Jul 17 22:10:54.409811 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.409696    2563 scope.go:117] "RemoveContainer" containerID="7223ee284c91905383e55f1f2e9eb2c2bbed625ccfaf5f399d0a3449dd7c25c1"
Jul 17 22:10:54.709959 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.709922    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"]
Jul 17 22:10:54.713943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.713916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.716940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.716922    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-539e6669\"/\"kube-root-ca.crt\""
Jul 17 22:10:54.717784 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.717771    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-539e6669\"/\"e2e-lifecycle-model-539e6669-kserve-self-signed-certs\""
Jul 17 22:10:54.717907 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.717888    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-539e6669\"/\"default-dockercfg-m2qms\""
Jul 17 22:10:54.717976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.717956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-539e6669\"/\"openshift-service-ca.crt\""
Jul 17 22:10:54.726270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.726247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"]
Jul 17 22:10:54.747082 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.747048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-home\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.747082 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.747082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tmp-dir\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.747258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.747106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tls-certs\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.747258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.747135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcrv\" (UniqueName: \"kubernetes.io/projected/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-kube-api-access-xdcrv\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.747258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.747172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-dshm\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.747258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.747232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-model-cache\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.847879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.847778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-home\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.847879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.847819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tmp-dir\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.847879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.847844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tls-certs\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.847879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.847877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcrv\" (UniqueName: \"kubernetes.io/projected/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-kube-api-access-xdcrv\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.848231 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.847901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-dshm\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.848231 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.847975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-model-cache\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.848350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.848279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-home\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.848350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.848317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tmp-dir\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.848479 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.848358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-model-cache\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.850331 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.850311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-dshm\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.850575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.850559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tls-certs\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:54.862986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:54.862962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcrv\" (UniqueName: \"kubernetes.io/projected/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-kube-api-access-xdcrv\") pod \"e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") " pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:55.023619 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.023566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:10:55.165375 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.165348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"]
Jul 17 22:10:55.204430 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:10:55.204397    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8fc4c99_4eab_4a47_aac9_9ebea5f2ba6d.slice/crio-00567084aac545401e4b8528579a234c2d50bae375110734975fa3fb390c2703 WatchSource:0}: Error finding container 00567084aac545401e4b8528579a234c2d50bae375110734975fa3fb390c2703: Status 404 returned error can't find the container with id 00567084aac545401e4b8528579a234c2d50bae375110734975fa3fb390c2703
Jul 17 22:10:55.415990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.415946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25" event={"ID":"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d","Type":"ContainerStarted","Data":"d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d"}
Jul 17 22:10:55.416467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.415997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25" event={"ID":"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d","Type":"ContainerStarted","Data":"00567084aac545401e4b8528579a234c2d50bae375110734975fa3fb390c2703"}
Jul 17 22:10:55.417827 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.417802    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerID="89e9e6c32311136cba879d3a8986fb667a5baecaf022a33521ed3f38e19913b0" exitCode=6
Jul 17 22:10:55.417947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.417855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" event={"ID":"e9a6addd-d5bd-4864-b8ff-95dd65f03e16","Type":"ContainerDied","Data":"89e9e6c32311136cba879d3a8986fb667a5baecaf022a33521ed3f38e19913b0"}
Jul 17 22:10:55.417947 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.417906    2563 scope.go:117] "RemoveContainer" containerID="7223ee284c91905383e55f1f2e9eb2c2bbed625ccfaf5f399d0a3449dd7c25c1"
Jul 17 22:10:55.418051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.418029    2563 scope.go:117] "RemoveContainer" containerID="89e9e6c32311136cba879d3a8986fb667a5baecaf022a33521ed3f38e19913b0"
Jul 17 22:10:55.418260 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:55.418218    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2_odh-ai-gateway-infra(e9a6addd-d5bd-4864-b8ff-95dd65f03e16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16"
Jul 17 22:10:55.451185 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:55.451134    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25" podStartSLOduration=1.451117672 podStartE2EDuration="1.451117672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:55.43407184 +0000 UTC m=+2619.346312129" watchObservedRunningTime="2026-07-17 22:10:55.451117672 +0000 UTC m=+2619.363357973"
Jul 17 22:10:56.426081 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:56.426052    2563 scope.go:117] "RemoveContainer" containerID="89e9e6c32311136cba879d3a8986fb667a5baecaf022a33521ed3f38e19913b0"
Jul 17 22:10:56.426535 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:10:56.426284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2_odh-ai-gateway-infra(e9a6addd-d5bd-4864-b8ff-95dd65f03e16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16"
Jul 17 22:10:59.978218 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:59.978179    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"]
Jul 17 22:10:59.982897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:59.982879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:10:59.990167 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:10:59.990060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"]
Jul 17 22:11:00.102996 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:00.102963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b560e3c-2eb2-4467-99a5-690872b237a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9\" (UID: \"0b560e3c-2eb2-4467-99a5-690872b237a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:11:00.203496 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:00.203457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b560e3c-2eb2-4467-99a5-690872b237a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9\" (UID: \"0b560e3c-2eb2-4467-99a5-690872b237a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:11:00.204071 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:00.204052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b560e3c-2eb2-4467-99a5-690872b237a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9\" (UID: \"0b560e3c-2eb2-4467-99a5-690872b237a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:11:00.295164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:00.295087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:11:00.433933 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:00.433911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"]
Jul 17 22:11:00.435622 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:00.435575    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b560e3c_2eb2_4467_99a5_690872b237a0.slice/crio-6470b0354279de82f31ec4e3d98921b79fea9f2b83613e10346b57f2f9e8901d WatchSource:0}: Error finding container 6470b0354279de82f31ec4e3d98921b79fea9f2b83613e10346b57f2f9e8901d: Status 404 returned error can't find the container with id 6470b0354279de82f31ec4e3d98921b79fea9f2b83613e10346b57f2f9e8901d
Jul 17 22:11:00.448140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:00.448111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" event={"ID":"0b560e3c-2eb2-4467-99a5-690872b237a0","Type":"ContainerStarted","Data":"6470b0354279de82f31ec4e3d98921b79fea9f2b83613e10346b57f2f9e8901d"}
Jul 17 22:11:01.454839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:01.454800    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerID="c1d598605fda2cdce5bb47bb67eaea021ab6a471d4c9e5d36603e781de3cb27a" exitCode=6
Jul 17 22:11:01.455233 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:01.454850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" event={"ID":"0b560e3c-2eb2-4467-99a5-690872b237a0","Type":"ContainerDied","Data":"c1d598605fda2cdce5bb47bb67eaea021ab6a471d4c9e5d36603e781de3cb27a"}
Jul 17 22:11:01.455233 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:01.455046    2563 scope.go:117] "RemoveContainer" containerID="c1d598605fda2cdce5bb47bb67eaea021ab6a471d4c9e5d36603e781de3cb27a"
Jul 17 22:11:01.596471 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:01.596439    2563 scope.go:117] "RemoveContainer" containerID="a7e85aa28bd04cb8416418b23e17a814a0b60e278539fb65e07a9993349a5de7"
Jul 17 22:11:02.462135 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.462102    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerID="78c39f55d0163b7f73f7a7e6cb5c03785759c8e4a896962febc1d7945ce51f28" exitCode=6
Jul 17 22:11:02.462645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.462187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" event={"ID":"0b560e3c-2eb2-4467-99a5-690872b237a0","Type":"ContainerDied","Data":"78c39f55d0163b7f73f7a7e6cb5c03785759c8e4a896962febc1d7945ce51f28"}
Jul 17 22:11:02.462645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.462246    2563 scope.go:117] "RemoveContainer" containerID="c1d598605fda2cdce5bb47bb67eaea021ab6a471d4c9e5d36603e781de3cb27a"
Jul 17 22:11:02.462645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.462439    2563 scope.go:117] "RemoveContainer" containerID="78c39f55d0163b7f73f7a7e6cb5c03785759c8e4a896962febc1d7945ce51f28"
Jul 17 22:11:02.462804 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:02.462680    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9_odh-ai-gateway-infra(0b560e3c-2eb2-4467-99a5-690872b237a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0"
Jul 17 22:11:02.464066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.464044    2563 generic.go:356] "Generic (PLEG): container finished" podID="6387c744-c265-44e6-924a-e779f9e299dc" containerID="b04677c15abdfaef780988b2b542aac625aceb32d9c05a10f1123711faf420b1" exitCode=6
Jul 17 22:11:02.464136 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.464110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" event={"ID":"6387c744-c265-44e6-924a-e779f9e299dc","Type":"ContainerDied","Data":"b04677c15abdfaef780988b2b542aac625aceb32d9c05a10f1123711faf420b1"}
Jul 17 22:11:02.464360 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.464347    2563 scope.go:117] "RemoveContainer" containerID="b04677c15abdfaef780988b2b542aac625aceb32d9c05a10f1123711faf420b1"
Jul 17 22:11:02.464532 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:02.464516    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp_odh-ai-gateway-infra(6387c744-c265-44e6-924a-e779f9e299dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" podUID="6387c744-c265-44e6-924a-e779f9e299dc"
Jul 17 22:11:02.473840 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:02.473816    2563 scope.go:117] "RemoveContainer" containerID="a7e85aa28bd04cb8416418b23e17a814a0b60e278539fb65e07a9993349a5de7"
Jul 17 22:11:03.471673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:03.471648    2563 scope.go:117] "RemoveContainer" containerID="78c39f55d0163b7f73f7a7e6cb5c03785759c8e4a896962febc1d7945ce51f28"
Jul 17 22:11:03.472052 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:03.471860    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9_odh-ai-gateway-infra(0b560e3c-2eb2-4467-99a5-690872b237a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0"
Jul 17 22:11:03.503291 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:03.503261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"]
Jul 17 22:11:03.633346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:03.633320    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:11:03.736392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:03.736302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6387c744-c265-44e6-924a-e779f9e299dc-maas-api-service-ca\") pod \"6387c744-c265-44e6-924a-e779f9e299dc\" (UID: \"6387c744-c265-44e6-924a-e779f9e299dc\") "
Jul 17 22:11:03.736698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:03.736676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6387c744-c265-44e6-924a-e779f9e299dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6387c744-c265-44e6-924a-e779f9e299dc" (UID: "6387c744-c265-44e6-924a-e779f9e299dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:03.837287 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:03.837256    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6387c744-c265-44e6-924a-e779f9e299dc-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:04.477388 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:04.477357    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"
Jul 17 22:11:04.477845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:04.477384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp" event={"ID":"6387c744-c265-44e6-924a-e779f9e299dc","Type":"ContainerDied","Data":"bd957f39507aedb38d85fbcc9e4a3c10389b736261a4ddaa0aaa2af80d515cc1"}
Jul 17 22:11:04.477845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:04.477448    2563 scope.go:117] "RemoveContainer" containerID="b04677c15abdfaef780988b2b542aac625aceb32d9c05a10f1123711faf420b1"
Jul 17 22:11:04.499200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:04.499168    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"]
Jul 17 22:11:04.501208 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:04.501189    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-dhlcp"]
Jul 17 22:11:04.603252 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:04.603218    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6387c744-c265-44e6-924a-e779f9e299dc" path="/var/lib/kubelet/pods/6387c744-c265-44e6-924a-e779f9e299dc/volumes"
Jul 17 22:11:05.023808 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:05.023774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:11:05.024040 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:05.024001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:11:05.036577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:05.036546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:11:05.494746 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:05.494712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:11:07.794961 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.794917    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"]
Jul 17 22:11:07.795481 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795460    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795485    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795527    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795544    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795858 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795858 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.795858 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.795698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6387c744-c265-44e6-924a-e779f9e299dc" containerName="revoke-keys"
Jul 17 22:11:07.800353 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.800331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:07.805086 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.805051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"]
Jul 17 22:11:07.876087 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.876057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2056f518-7428-4524-8c3b-f5187a3f4b5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5\" (UID: \"2056f518-7428-4524-8c3b-f5187a3f4b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:07.977360 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.977326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2056f518-7428-4524-8c3b-f5187a3f4b5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5\" (UID: \"2056f518-7428-4524-8c3b-f5187a3f4b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:07.978021 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:07.978001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2056f518-7428-4524-8c3b-f5187a3f4b5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5\" (UID: \"2056f518-7428-4524-8c3b-f5187a3f4b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:08.111974 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:08.111897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:08.271616 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:08.271570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"]
Jul 17 22:11:08.273488 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:08.273455    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2056f518_7428_4524_8c3b_f5187a3f4b5f.slice/crio-cea99cb6abc4d7cb38f198ba4019f15b852eeb3eabdd4c7ff7f5ce042c7b2abb WatchSource:0}: Error finding container cea99cb6abc4d7cb38f198ba4019f15b852eeb3eabdd4c7ff7f5ce042c7b2abb: Status 404 returned error can't find the container with id cea99cb6abc4d7cb38f198ba4019f15b852eeb3eabdd4c7ff7f5ce042c7b2abb
Jul 17 22:11:08.500090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:08.500058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" event={"ID":"2056f518-7428-4524-8c3b-f5187a3f4b5f","Type":"ContainerStarted","Data":"cea99cb6abc4d7cb38f198ba4019f15b852eeb3eabdd4c7ff7f5ce042c7b2abb"}
Jul 17 22:11:09.506979 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:09.506946    2563 generic.go:356] "Generic (PLEG): container finished" podID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerID="8add137cb5534b84546b8d9b88a2aa1a2660cf69e4034690cad71bae7b0fff0d" exitCode=6
Jul 17 22:11:09.507407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:09.507037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" event={"ID":"2056f518-7428-4524-8c3b-f5187a3f4b5f","Type":"ContainerDied","Data":"8add137cb5534b84546b8d9b88a2aa1a2660cf69e4034690cad71bae7b0fff0d"}
Jul 17 22:11:09.507407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:09.507197    2563 scope.go:117] "RemoveContainer" containerID="8add137cb5534b84546b8d9b88a2aa1a2660cf69e4034690cad71bae7b0fff0d"
Jul 17 22:11:09.596853 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:09.596821    2563 scope.go:117] "RemoveContainer" containerID="89e9e6c32311136cba879d3a8986fb667a5baecaf022a33521ed3f38e19913b0"
Jul 17 22:11:10.513774 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.513745    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerID="8947212f8de796a41b0834032e5295e2bd6d798da1f3c6b17fa6b0667195948e" exitCode=6
Jul 17 22:11:10.514215 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.513817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" event={"ID":"e9a6addd-d5bd-4864-b8ff-95dd65f03e16","Type":"ContainerDied","Data":"8947212f8de796a41b0834032e5295e2bd6d798da1f3c6b17fa6b0667195948e"}
Jul 17 22:11:10.514215 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.513872    2563 scope.go:117] "RemoveContainer" containerID="89e9e6c32311136cba879d3a8986fb667a5baecaf022a33521ed3f38e19913b0"
Jul 17 22:11:10.514215 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.514195    2563 scope.go:117] "RemoveContainer" containerID="8947212f8de796a41b0834032e5295e2bd6d798da1f3c6b17fa6b0667195948e"
Jul 17 22:11:10.514410 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:10.514385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2_odh-ai-gateway-infra(e9a6addd-d5bd-4864-b8ff-95dd65f03e16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16"
Jul 17 22:11:10.515836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.515814    2563 generic.go:356] "Generic (PLEG): container finished" podID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerID="e62c61d2231d0bedd4483902045b49e40fe34fb9e5e53b4f8e27d20866a7a7eb" exitCode=6
Jul 17 22:11:10.515932 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.515881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" event={"ID":"2056f518-7428-4524-8c3b-f5187a3f4b5f","Type":"ContainerDied","Data":"e62c61d2231d0bedd4483902045b49e40fe34fb9e5e53b4f8e27d20866a7a7eb"}
Jul 17 22:11:10.516090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.516077    2563 scope.go:117] "RemoveContainer" containerID="e62c61d2231d0bedd4483902045b49e40fe34fb9e5e53b4f8e27d20866a7a7eb"
Jul 17 22:11:10.516264 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:10.516241    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5_odh-ai-gateway-infra(2056f518-7428-4524-8c3b-f5187a3f4b5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f"
Jul 17 22:11:10.525403 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:10.525380    2563 scope.go:117] "RemoveContainer" containerID="8add137cb5534b84546b8d9b88a2aa1a2660cf69e4034690cad71bae7b0fff0d"
Jul 17 22:11:11.523792 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:11.523766    2563 scope.go:117] "RemoveContainer" containerID="e62c61d2231d0bedd4483902045b49e40fe34fb9e5e53b4f8e27d20866a7a7eb"
Jul 17 22:11:11.524153 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:11.523956    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5_odh-ai-gateway-infra(2056f518-7428-4524-8c3b-f5187a3f4b5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f"
Jul 17 22:11:11.544568 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:11.544535    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"]
Jul 17 22:11:11.675029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:11.675005    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:11:11.810913 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:11.810827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a6addd-d5bd-4864-b8ff-95dd65f03e16-maas-api-service-ca\") pod \"e9a6addd-d5bd-4864-b8ff-95dd65f03e16\" (UID: \"e9a6addd-d5bd-4864-b8ff-95dd65f03e16\") "
Jul 17 22:11:11.811204 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:11.811182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a6addd-d5bd-4864-b8ff-95dd65f03e16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9a6addd-d5bd-4864-b8ff-95dd65f03e16" (UID: "e9a6addd-d5bd-4864-b8ff-95dd65f03e16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:11.912254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:11.912205    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a6addd-d5bd-4864-b8ff-95dd65f03e16-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:12.529703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:12.529670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"
Jul 17 22:11:12.529703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:12.529683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2" event={"ID":"e9a6addd-d5bd-4864-b8ff-95dd65f03e16","Type":"ContainerDied","Data":"a00ee3dac71bf22d1cb6089572650effb47bbc3500dcd065e5a1f34cb606dc19"}
Jul 17 22:11:12.530190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:12.529746    2563 scope.go:117] "RemoveContainer" containerID="8947212f8de796a41b0834032e5295e2bd6d798da1f3c6b17fa6b0667195948e"
Jul 17 22:11:12.549486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:12.549462    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"]
Jul 17 22:11:12.555543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:12.555520    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-f77c2"]
Jul 17 22:11:12.602986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:12.602956    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" path="/var/lib/kubelet/pods/e9a6addd-d5bd-4864-b8ff-95dd65f03e16/volumes"
Jul 17 22:11:15.597393 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:15.597364    2563 scope.go:117] "RemoveContainer" containerID="78c39f55d0163b7f73f7a7e6cb5c03785759c8e4a896962febc1d7945ce51f28"
Jul 17 22:11:16.551696 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:16.551663    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerID="52977590ead100ffa3ad83044b27ffe437ae763b6dd6b1640cb9c9408a523dd9" exitCode=6
Jul 17 22:11:16.551866 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:16.551731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" event={"ID":"0b560e3c-2eb2-4467-99a5-690872b237a0","Type":"ContainerDied","Data":"52977590ead100ffa3ad83044b27ffe437ae763b6dd6b1640cb9c9408a523dd9"}
Jul 17 22:11:16.551866 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:16.551814    2563 scope.go:117] "RemoveContainer" containerID="78c39f55d0163b7f73f7a7e6cb5c03785759c8e4a896962febc1d7945ce51f28"
Jul 17 22:11:16.552069 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:16.552054    2563 scope.go:117] "RemoveContainer" containerID="52977590ead100ffa3ad83044b27ffe437ae763b6dd6b1640cb9c9408a523dd9"
Jul 17 22:11:16.552285 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:16.552267    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9_odh-ai-gateway-infra(0b560e3c-2eb2-4467-99a5-690872b237a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0"
Jul 17 22:11:17.576498 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:17.576456    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"]
Jul 17 22:11:17.719533 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:17.719507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:11:17.863508 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:17.863407    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b560e3c-2eb2-4467-99a5-690872b237a0-maas-api-service-ca\") pod \"0b560e3c-2eb2-4467-99a5-690872b237a0\" (UID: \"0b560e3c-2eb2-4467-99a5-690872b237a0\") "
Jul 17 22:11:17.863777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:17.863753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b560e3c-2eb2-4467-99a5-690872b237a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b560e3c-2eb2-4467-99a5-690872b237a0" (UID: "0b560e3c-2eb2-4467-99a5-690872b237a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:17.964837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:17.964805    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b560e3c-2eb2-4467-99a5-690872b237a0-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:18.565221 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:18.565186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9" event={"ID":"0b560e3c-2eb2-4467-99a5-690872b237a0","Type":"ContainerDied","Data":"6470b0354279de82f31ec4e3d98921b79fea9f2b83613e10346b57f2f9e8901d"}
Jul 17 22:11:18.565221 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:18.565219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"
Jul 17 22:11:18.565488 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:18.565242    2563 scope.go:117] "RemoveContainer" containerID="52977590ead100ffa3ad83044b27ffe437ae763b6dd6b1640cb9c9408a523dd9"
Jul 17 22:11:18.587436 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:18.587409    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"]
Jul 17 22:11:18.593660 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:18.593639    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qrjv9"]
Jul 17 22:11:18.603511 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:18.603488    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" path="/var/lib/kubelet/pods/0b560e3c-2eb2-4467-99a5-690872b237a0/volumes"
Jul 17 22:11:21.798240 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798207    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"]
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798574    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798584    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798611    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798619    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798653    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798661    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798666    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.798682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798671    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.799028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798737    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:21.799028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798746    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.799028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798752    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.799028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798758    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b560e3c-2eb2-4467-99a5-690872b237a0" containerName="revoke-keys"
Jul 17 22:11:21.799028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.798764    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:21.803410 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.803393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:21.807741 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.807712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"]
Jul 17 22:11:21.999509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:21.999474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7042b7-f051-498e-8669-0b79ef23fa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-539e6669-kntrx\" (UID: \"bd7042b7-f051-498e-8669-0b79ef23fa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:22.100565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.100481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7042b7-f051-498e-8669-0b79ef23fa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-539e6669-kntrx\" (UID: \"bd7042b7-f051-498e-8669-0b79ef23fa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:22.101113 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.101090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7042b7-f051-498e-8669-0b79ef23fa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-539e6669-kntrx\" (UID: \"bd7042b7-f051-498e-8669-0b79ef23fa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:22.117464 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.117438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:22.253277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.253192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"]
Jul 17 22:11:22.256739 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:22.256700    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7042b7_f051_498e_8669_0b79ef23fa30.slice/crio-dc2d2330f3346c2eadbc37387c6f42386c62d0c765f06730dc6fbaa9bcb45852 WatchSource:0}: Error finding container dc2d2330f3346c2eadbc37387c6f42386c62d0c765f06730dc6fbaa9bcb45852: Status 404 returned error can't find the container with id dc2d2330f3346c2eadbc37387c6f42386c62d0c765f06730dc6fbaa9bcb45852
Jul 17 22:11:22.588416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.588385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx" event={"ID":"bd7042b7-f051-498e-8669-0b79ef23fa30","Type":"ContainerStarted","Data":"de2899e55fb46930df0b2f9eafc8667cc620dff7dc4fbc09dd0c80ab8058832f"}
Jul 17 22:11:22.588565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.588437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx" event={"ID":"bd7042b7-f051-498e-8669-0b79ef23fa30","Type":"ContainerStarted","Data":"dc2d2330f3346c2eadbc37387c6f42386c62d0c765f06730dc6fbaa9bcb45852"}
Jul 17 22:11:22.603369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:22.603320    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx" podStartSLOduration=1.603304304 podStartE2EDuration="1.603304304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:22.602092507 +0000 UTC m=+2646.514332797" watchObservedRunningTime="2026-07-17 22:11:22.603304304 +0000 UTC m=+2646.515544623"
Jul 17 22:11:23.594801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:23.594770    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd7042b7-f051-498e-8669-0b79ef23fa30" containerID="de2899e55fb46930df0b2f9eafc8667cc620dff7dc4fbc09dd0c80ab8058832f" exitCode=0
Jul 17 22:11:23.595211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:23.594865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx" event={"ID":"bd7042b7-f051-498e-8669-0b79ef23fa30","Type":"ContainerDied","Data":"de2899e55fb46930df0b2f9eafc8667cc620dff7dc4fbc09dd0c80ab8058832f"}
Jul 17 22:11:24.597140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:24.597114    2563 scope.go:117] "RemoveContainer" containerID="e62c61d2231d0bedd4483902045b49e40fe34fb9e5e53b4f8e27d20866a7a7eb"
Jul 17 22:11:24.748658 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:24.748629    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:24.822488 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:24.822459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7042b7-f051-498e-8669-0b79ef23fa30-maas-api-service-ca\") pod \"bd7042b7-f051-498e-8669-0b79ef23fa30\" (UID: \"bd7042b7-f051-498e-8669-0b79ef23fa30\") "
Jul 17 22:11:24.822865 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:24.822830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7042b7-f051-498e-8669-0b79ef23fa30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd7042b7-f051-498e-8669-0b79ef23fa30" (UID: "bd7042b7-f051-498e-8669-0b79ef23fa30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:24.923982 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:24.923951    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7042b7-f051-498e-8669-0b79ef23fa30-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:25.608181 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.608153    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx"
Jul 17 22:11:25.608637 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.608150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-539e6669-kntrx" event={"ID":"bd7042b7-f051-498e-8669-0b79ef23fa30","Type":"ContainerDied","Data":"dc2d2330f3346c2eadbc37387c6f42386c62d0c765f06730dc6fbaa9bcb45852"}
Jul 17 22:11:25.608637 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.608296    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2d2330f3346c2eadbc37387c6f42386c62d0c765f06730dc6fbaa9bcb45852"
Jul 17 22:11:25.610033 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.610009    2563 generic.go:356] "Generic (PLEG): container finished" podID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerID="480c7722d9e98fd346dec31567bd144165e46a0cc5676fd1333f4a5b720b6691" exitCode=6
Jul 17 22:11:25.610164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.610040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" event={"ID":"2056f518-7428-4524-8c3b-f5187a3f4b5f","Type":"ContainerDied","Data":"480c7722d9e98fd346dec31567bd144165e46a0cc5676fd1333f4a5b720b6691"}
Jul 17 22:11:25.610164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.610092    2563 scope.go:117] "RemoveContainer" containerID="e62c61d2231d0bedd4483902045b49e40fe34fb9e5e53b4f8e27d20866a7a7eb"
Jul 17 22:11:25.610394 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:25.610375    2563 scope.go:117] "RemoveContainer" containerID="480c7722d9e98fd346dec31567bd144165e46a0cc5676fd1333f4a5b720b6691"
Jul 17 22:11:25.610583 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:25.610566    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5_odh-ai-gateway-infra(2056f518-7428-4524-8c3b-f5187a3f4b5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f"
Jul 17 22:11:26.634528 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:26.634495    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"]
Jul 17 22:11:26.775089 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:26.775062    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:26.841255 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:26.841217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2056f518-7428-4524-8c3b-f5187a3f4b5f-maas-api-service-ca\") pod \"2056f518-7428-4524-8c3b-f5187a3f4b5f\" (UID: \"2056f518-7428-4524-8c3b-f5187a3f4b5f\") "
Jul 17 22:11:26.841625 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:26.841575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2056f518-7428-4524-8c3b-f5187a3f4b5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2056f518-7428-4524-8c3b-f5187a3f4b5f" (UID: "2056f518-7428-4524-8c3b-f5187a3f4b5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:26.942245 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:26.942209    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2056f518-7428-4524-8c3b-f5187a3f4b5f-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:27.622503 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:27.622469    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"
Jul 17 22:11:27.622825 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:27.622519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5" event={"ID":"2056f518-7428-4524-8c3b-f5187a3f4b5f","Type":"ContainerDied","Data":"cea99cb6abc4d7cb38f198ba4019f15b852eeb3eabdd4c7ff7f5ce042c7b2abb"}
Jul 17 22:11:27.622825 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:27.622576    2563 scope.go:117] "RemoveContainer" containerID="480c7722d9e98fd346dec31567bd144165e46a0cc5676fd1333f4a5b720b6691"
Jul 17 22:11:27.645174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:27.645149    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"]
Jul 17 22:11:27.650940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:27.650919    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-k55b5"]
Jul 17 22:11:28.602620 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:28.602568    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" path="/var/lib/kubelet/pods/2056f518-7428-4524-8c3b-f5187a3f4b5f/volumes"
Jul 17 22:11:35.708468 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708429    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"]
Jul 17 22:11:35.708879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708840    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:35.708879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708854    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:35.708879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708866    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:35.708879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708872    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708883    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7042b7-f051-498e-8669-0b79ef23fa30" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708889    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7042b7-f051-498e-8669-0b79ef23fa30" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708954    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7042b7-f051-498e-8669-0b79ef23fa30" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708963    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708969    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a6addd-d5bd-4864-b8ff-95dd65f03e16" containerName="revoke-keys"
Jul 17 22:11:35.709014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.708977    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:35.713575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.713558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:35.716112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.716092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:11:35.719756 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.719551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"]
Jul 17 22:11:35.820810 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.820773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8ef13c-1ecc-4458-9591-044e53e6e28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz\" (UID: \"7c8ef13c-1ecc-4458-9591-044e53e6e28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:35.922111 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.922070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8ef13c-1ecc-4458-9591-044e53e6e28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz\" (UID: \"7c8ef13c-1ecc-4458-9591-044e53e6e28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:35.922706 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:35.922688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8ef13c-1ecc-4458-9591-044e53e6e28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz\" (UID: \"7c8ef13c-1ecc-4458-9591-044e53e6e28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:36.027211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:36.027110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:36.163466 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:36.163443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"]
Jul 17 22:11:36.166430 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:36.166398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8ef13c_1ecc_4458_9591_044e53e6e28a.slice/crio-01987abed767e7cfe3ffe00a6c9a0536d69f6db1dcf022a43ac730447fea5345 WatchSource:0}: Error finding container 01987abed767e7cfe3ffe00a6c9a0536d69f6db1dcf022a43ac730447fea5345: Status 404 returned error can't find the container with id 01987abed767e7cfe3ffe00a6c9a0536d69f6db1dcf022a43ac730447fea5345
Jul 17 22:11:36.667063 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:36.667029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" event={"ID":"7c8ef13c-1ecc-4458-9591-044e53e6e28a","Type":"ContainerStarted","Data":"00fa592461e170bc9771d8bd54ceda614269ad4baa1a97a949eb3c7d6b4130b6"}
Jul 17 22:11:36.667250 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:36.667090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" event={"ID":"7c8ef13c-1ecc-4458-9591-044e53e6e28a","Type":"ContainerStarted","Data":"01987abed767e7cfe3ffe00a6c9a0536d69f6db1dcf022a43ac730447fea5345"}
Jul 17 22:11:36.681352 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:36.681302    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" podStartSLOduration=1.681286728 podStartE2EDuration="1.681286728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:36.680556537 +0000 UTC m=+2660.592796858" watchObservedRunningTime="2026-07-17 22:11:36.681286728 +0000 UTC m=+2660.593527038"
Jul 17 22:11:37.673207 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:37.673170    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerID="00fa592461e170bc9771d8bd54ceda614269ad4baa1a97a949eb3c7d6b4130b6" exitCode=6
Jul 17 22:11:37.673669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:37.673251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" event={"ID":"7c8ef13c-1ecc-4458-9591-044e53e6e28a","Type":"ContainerDied","Data":"00fa592461e170bc9771d8bd54ceda614269ad4baa1a97a949eb3c7d6b4130b6"}
Jul 17 22:11:37.673669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:37.673464    2563 scope.go:117] "RemoveContainer" containerID="00fa592461e170bc9771d8bd54ceda614269ad4baa1a97a949eb3c7d6b4130b6"
Jul 17 22:11:38.680310 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:38.680275    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerID="f315cc013d27db309d69376f891cfd35f5fe085dd16a689fabf24cff0849efa5" exitCode=6
Jul 17 22:11:38.680831 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:38.680362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" event={"ID":"7c8ef13c-1ecc-4458-9591-044e53e6e28a","Type":"ContainerDied","Data":"f315cc013d27db309d69376f891cfd35f5fe085dd16a689fabf24cff0849efa5"}
Jul 17 22:11:38.680831 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:38.680434    2563 scope.go:117] "RemoveContainer" containerID="00fa592461e170bc9771d8bd54ceda614269ad4baa1a97a949eb3c7d6b4130b6"
Jul 17 22:11:38.680831 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:38.680536    2563 scope.go:117] "RemoveContainer" containerID="f315cc013d27db309d69376f891cfd35f5fe085dd16a689fabf24cff0849efa5"
Jul 17 22:11:38.680831 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:38.680780    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz_odh-ai-gateway-infra(7c8ef13c-1ecc-4458-9591-044e53e6e28a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a"
Jul 17 22:11:39.687467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:39.687441    2563 scope.go:117] "RemoveContainer" containerID="f315cc013d27db309d69376f891cfd35f5fe085dd16a689fabf24cff0849efa5"
Jul 17 22:11:39.687938 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:39.687719    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz_odh-ai-gateway-infra(7c8ef13c-1ecc-4458-9591-044e53e6e28a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a"
Jul 17 22:11:41.868554 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:41.868514    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"]
Jul 17 22:11:43.169114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.169077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"]
Jul 17 22:11:43.169669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.169649    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:43.169768 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.169671    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:43.169830 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.169786    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2056f518-7428-4524-8c3b-f5187a3f4b5f" containerName="revoke-keys"
Jul 17 22:11:43.174077 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.174056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:11:43.178470 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.178446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"]
Jul 17 22:11:43.284194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.284160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3332b4-eedf-4151-8fad-91b114d6f907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-r64st\" (UID: \"6b3332b4-eedf-4151-8fad-91b114d6f907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:11:43.385338 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.385304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3332b4-eedf-4151-8fad-91b114d6f907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-r64st\" (UID: \"6b3332b4-eedf-4151-8fad-91b114d6f907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:11:43.385952 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.385931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3332b4-eedf-4151-8fad-91b114d6f907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-r64st\" (UID: \"6b3332b4-eedf-4151-8fad-91b114d6f907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:11:43.485819 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.485743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:11:43.619960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.619924    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"]
Jul 17 22:11:43.624149 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:43.624117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3332b4_eedf_4151_8fad_91b114d6f907.slice/crio-b2ea3708e56fa69cb9bf08c71467270de88c78fc9a6630c59afd25bf463202f4 WatchSource:0}: Error finding container b2ea3708e56fa69cb9bf08c71467270de88c78fc9a6630c59afd25bf463202f4: Status 404 returned error can't find the container with id b2ea3708e56fa69cb9bf08c71467270de88c78fc9a6630c59afd25bf463202f4
Jul 17 22:11:43.711213 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:43.711183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" event={"ID":"6b3332b4-eedf-4151-8fad-91b114d6f907","Type":"ContainerStarted","Data":"b2ea3708e56fa69cb9bf08c71467270de88c78fc9a6630c59afd25bf463202f4"}
Jul 17 22:11:44.717776 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:44.717736    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerID="b7e58f1f533a3f81eb88876bd059605f85692164b00b3447b09537c5d89ab3ac" exitCode=6
Jul 17 22:11:44.718183 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:44.717825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" event={"ID":"6b3332b4-eedf-4151-8fad-91b114d6f907","Type":"ContainerDied","Data":"b7e58f1f533a3f81eb88876bd059605f85692164b00b3447b09537c5d89ab3ac"}
Jul 17 22:11:44.718183 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:44.717989    2563 scope.go:117] "RemoveContainer" containerID="b7e58f1f533a3f81eb88876bd059605f85692164b00b3447b09537c5d89ab3ac"
Jul 17 22:11:45.724484 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:45.724445    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerID="4208de54cfc96b5f2d5cbc11c853cbd6b0c249d7683b3d9c98b4f2ef9d5e56a6" exitCode=6
Jul 17 22:11:45.724960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:45.724528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" event={"ID":"6b3332b4-eedf-4151-8fad-91b114d6f907","Type":"ContainerDied","Data":"4208de54cfc96b5f2d5cbc11c853cbd6b0c249d7683b3d9c98b4f2ef9d5e56a6"}
Jul 17 22:11:45.724960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:45.724583    2563 scope.go:117] "RemoveContainer" containerID="b7e58f1f533a3f81eb88876bd059605f85692164b00b3447b09537c5d89ab3ac"
Jul 17 22:11:45.724960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:45.724821    2563 scope.go:117] "RemoveContainer" containerID="4208de54cfc96b5f2d5cbc11c853cbd6b0c249d7683b3d9c98b4f2ef9d5e56a6"
Jul 17 22:11:45.725117 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:45.725053    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-r64st_odh-ai-gateway-infra(6b3332b4-eedf-4151-8fad-91b114d6f907)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907"
Jul 17 22:11:46.730779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:46.730751    2563 scope.go:117] "RemoveContainer" containerID="4208de54cfc96b5f2d5cbc11c853cbd6b0c249d7683b3d9c98b4f2ef9d5e56a6"
Jul 17 22:11:46.731172 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:46.730923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-r64st_odh-ai-gateway-infra(6b3332b4-eedf-4151-8fad-91b114d6f907)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907"
Jul 17 22:11:50.018118 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.018081    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"]
Jul 17 22:11:50.022937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.022913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:11:50.028390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.028340    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"]
Jul 17 22:11:50.144662 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.144592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a027125-34a9-41c6-810d-0e5831ba98d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4\" (UID: \"4a027125-34a9-41c6-810d-0e5831ba98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:11:50.246122 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.246086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a027125-34a9-41c6-810d-0e5831ba98d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4\" (UID: \"4a027125-34a9-41c6-810d-0e5831ba98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:11:50.246807 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.246788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a027125-34a9-41c6-810d-0e5831ba98d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4\" (UID: \"4a027125-34a9-41c6-810d-0e5831ba98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:11:50.336251 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.336176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:11:50.473104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.473077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"]
Jul 17 22:11:50.474499 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:50.474472    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a027125_34a9_41c6_810d_0e5831ba98d5.slice/crio-60332bc6732f9d99c5dcf12b19290f4ccb7c9dc7c2d3b679d5076a8916615dd6 WatchSource:0}: Error finding container 60332bc6732f9d99c5dcf12b19290f4ccb7c9dc7c2d3b679d5076a8916615dd6: Status 404 returned error can't find the container with id 60332bc6732f9d99c5dcf12b19290f4ccb7c9dc7c2d3b679d5076a8916615dd6
Jul 17 22:11:50.750193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:50.750152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" event={"ID":"4a027125-34a9-41c6-810d-0e5831ba98d5","Type":"ContainerStarted","Data":"60332bc6732f9d99c5dcf12b19290f4ccb7c9dc7c2d3b679d5076a8916615dd6"}
Jul 17 22:11:51.597001 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:51.596970    2563 scope.go:117] "RemoveContainer" containerID="f315cc013d27db309d69376f891cfd35f5fe085dd16a689fabf24cff0849efa5"
Jul 17 22:11:51.757867 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:51.757834    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerID="9dbcca279a698d42b3b62f0ed8ff28fa2fbbf43a2844be53f61439ad6c4608a9" exitCode=6
Jul 17 22:11:51.758025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:51.757920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" event={"ID":"4a027125-34a9-41c6-810d-0e5831ba98d5","Type":"ContainerDied","Data":"9dbcca279a698d42b3b62f0ed8ff28fa2fbbf43a2844be53f61439ad6c4608a9"}
Jul 17 22:11:51.758095 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:51.758080    2563 scope.go:117] "RemoveContainer" containerID="9dbcca279a698d42b3b62f0ed8ff28fa2fbbf43a2844be53f61439ad6c4608a9"
Jul 17 22:11:52.764739 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.764648    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerID="83395494d260a21ecdad8c25e4fbbdccbab6bc86f94518029f6fd6f40a23b722" exitCode=6
Jul 17 22:11:52.765151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.764736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" event={"ID":"7c8ef13c-1ecc-4458-9591-044e53e6e28a","Type":"ContainerDied","Data":"83395494d260a21ecdad8c25e4fbbdccbab6bc86f94518029f6fd6f40a23b722"}
Jul 17 22:11:52.765151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.764801    2563 scope.go:117] "RemoveContainer" containerID="f315cc013d27db309d69376f891cfd35f5fe085dd16a689fabf24cff0849efa5"
Jul 17 22:11:52.765151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.765120    2563 scope.go:117] "RemoveContainer" containerID="83395494d260a21ecdad8c25e4fbbdccbab6bc86f94518029f6fd6f40a23b722"
Jul 17 22:11:52.765382 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:52.765362    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz_odh-ai-gateway-infra(7c8ef13c-1ecc-4458-9591-044e53e6e28a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a"
Jul 17 22:11:52.766668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.766648    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerID="a26d11ca429226a7ae2de12452a67a472d0d128c6adff804bf384318a24f0387" exitCode=6
Jul 17 22:11:52.766759 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.766721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" event={"ID":"4a027125-34a9-41c6-810d-0e5831ba98d5","Type":"ContainerDied","Data":"a26d11ca429226a7ae2de12452a67a472d0d128c6adff804bf384318a24f0387"}
Jul 17 22:11:52.766927 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.766916    2563 scope.go:117] "RemoveContainer" containerID="a26d11ca429226a7ae2de12452a67a472d0d128c6adff804bf384318a24f0387"
Jul 17 22:11:52.767104 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:52.767089    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4_odh-ai-gateway-infra(4a027125-34a9-41c6-810d-0e5831ba98d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5"
Jul 17 22:11:52.775968 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:52.775953    2563 scope.go:117] "RemoveContainer" containerID="9dbcca279a698d42b3b62f0ed8ff28fa2fbbf43a2844be53f61439ad6c4608a9"
Jul 17 22:11:53.774918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:53.774888    2563 scope.go:117] "RemoveContainer" containerID="a26d11ca429226a7ae2de12452a67a472d0d128c6adff804bf384318a24f0387"
Jul 17 22:11:53.775319 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:53.775076    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4_odh-ai-gateway-infra(4a027125-34a9-41c6-810d-0e5831ba98d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5"
Jul 17 22:11:53.789740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:53.789684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"]
Jul 17 22:11:53.930522 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:53.930496    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:54.079957 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.079871    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8ef13c-1ecc-4458-9591-044e53e6e28a-maas-api-service-ca\") pod \"7c8ef13c-1ecc-4458-9591-044e53e6e28a\" (UID: \"7c8ef13c-1ecc-4458-9591-044e53e6e28a\") "
Jul 17 22:11:54.080229 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.080207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8ef13c-1ecc-4458-9591-044e53e6e28a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c8ef13c-1ecc-4458-9591-044e53e6e28a" (UID: "7c8ef13c-1ecc-4458-9591-044e53e6e28a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:54.181072 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.181036    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8ef13c-1ecc-4458-9591-044e53e6e28a-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:54.781237 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.781200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz" event={"ID":"7c8ef13c-1ecc-4458-9591-044e53e6e28a","Type":"ContainerDied","Data":"01987abed767e7cfe3ffe00a6c9a0536d69f6db1dcf022a43ac730447fea5345"}
Jul 17 22:11:54.781686 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.781268    2563 scope.go:117] "RemoveContainer" containerID="83395494d260a21ecdad8c25e4fbbdccbab6bc86f94518029f6fd6f40a23b722"
Jul 17 22:11:54.781686 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.781209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"
Jul 17 22:11:54.800918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.800892    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"]
Jul 17 22:11:54.806616 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:54.806578    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tjrsz"]
Jul 17 22:11:56.604207 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:56.604174    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" path="/var/lib/kubelet/pods/7c8ef13c-1ecc-4458-9591-044e53e6e28a/volumes"
Jul 17 22:11:56.889190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:56.889107    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25" podUID="f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" containerName="main" containerID="cri-o://d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d" gracePeriod=45
Jul 17 22:11:57.149927 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.149860    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:11:57.311427 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tmp-dir\") pod \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") "
Jul 17 22:11:57.311427 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdcrv\" (UniqueName: \"kubernetes.io/projected/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-kube-api-access-xdcrv\") pod \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") "
Jul 17 22:11:57.311701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-dshm\") pod \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") "
Jul 17 22:11:57.311701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-model-cache\") pod \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") "
Jul 17 22:11:57.311701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tls-certs\") pod \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") "
Jul 17 22:11:57.311701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-home\") pod \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\" (UID: \"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d\") "
Jul 17 22:11:57.311923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" (UID: "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:11:57.311923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" (UID: "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:11:57.312014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.311913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-home" (OuterVolumeSpecName: "home") pod "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" (UID: "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:11:57.312014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.312002    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:57.312084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.312026    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:57.312084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.312047    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:57.313925 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.313898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-kube-api-access-xdcrv" (OuterVolumeSpecName: "kube-api-access-xdcrv") pod "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" (UID: "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d"). InnerVolumeSpecName "kube-api-access-xdcrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:11:57.313925 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.313916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-dshm" (OuterVolumeSpecName: "dshm") pod "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" (UID: "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:11:57.314056 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.313962    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" (UID: "f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:11:57.412837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.412806    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:57.412837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.412832    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:57.412837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.412843    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdcrv\" (UniqueName: \"kubernetes.io/projected/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d-kube-api-access-xdcrv\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:11:57.799159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.799067    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" containerID="d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d" exitCode=0
Jul 17 22:11:57.799159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.799110    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"
Jul 17 22:11:57.799159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.799148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25" event={"ID":"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d","Type":"ContainerDied","Data":"d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d"}
Jul 17 22:11:57.799739 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.799190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25" event={"ID":"f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d","Type":"ContainerDied","Data":"00567084aac545401e4b8528579a234c2d50bae375110734975fa3fb390c2703"}
Jul 17 22:11:57.799739 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.799208    2563 scope.go:117] "RemoveContainer" containerID="d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d"
Jul 17 22:11:57.808832 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.808815    2563 scope.go:117] "RemoveContainer" containerID="d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d"
Jul 17 22:11:57.809072 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:11:57.809053    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d\": container with ID starting with d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d not found: ID does not exist" containerID="d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d"
Jul 17 22:11:57.809140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.809085    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d"} err="failed to get container status \"d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d\": rpc error: code = NotFound desc = could not find container \"d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d\": container with ID starting with d698ffeeda252569db4561c9b53594cfc98f57e2e36775ac8df46ab17778d25d not found: ID does not exist"
Jul 17 22:11:57.819702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.819678    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"]
Jul 17 22:11:57.823329 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.823304    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-539e6669/e2e-lifecycle-model-539e6669-kserve-5c64dd486c-q7w25"]
Jul 17 22:11:57.834691 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.834671    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"]
Jul 17 22:11:57.835056 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" containerName="main"
Jul 17 22:11:57.835107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835059    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" containerName="main"
Jul 17 22:11:57.835107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835084    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835090    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835096    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835101    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835123    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835182    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" containerName="main"
Jul 17 22:11:57.835259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835194    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.835259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.835205    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:11:57.839695 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.839681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:11:57.844409 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.844389    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"]
Jul 17 22:11:57.918074 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:57.918041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f820-642e-45df-bd8d-6e770e976023-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw\" (UID: \"dfe8f820-642e-45df-bd8d-6e770e976023\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:11:58.019529 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.019496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f820-642e-45df-bd8d-6e770e976023-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw\" (UID: \"dfe8f820-642e-45df-bd8d-6e770e976023\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:11:58.020163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.020142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f820-642e-45df-bd8d-6e770e976023-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw\" (UID: \"dfe8f820-642e-45df-bd8d-6e770e976023\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:11:58.150941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.150857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:11:58.288867 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.288838    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"]
Jul 17 22:11:58.290956 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:11:58.290924    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe8f820_642e_45df_bd8d_6e770e976023.slice/crio-a976d6310fb656b7dfc1447d0fe2a0544e98302faa29a01010f42a443c6f20cc WatchSource:0}: Error finding container a976d6310fb656b7dfc1447d0fe2a0544e98302faa29a01010f42a443c6f20cc: Status 404 returned error can't find the container with id a976d6310fb656b7dfc1447d0fe2a0544e98302faa29a01010f42a443c6f20cc
Jul 17 22:11:58.604143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.604109    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d" path="/var/lib/kubelet/pods/f8fc4c99-4eab-4a47-aac9-9ebea5f2ba6d/volumes"
Jul 17 22:11:58.806814 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.806734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerStarted","Data":"b99290aade22d02e8d79294fd83973ebf6ecb797ccc3217e4272da61f154e483"}
Jul 17 22:11:58.806814 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.806783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerStarted","Data":"a976d6310fb656b7dfc1447d0fe2a0544e98302faa29a01010f42a443c6f20cc"}
Jul 17 22:11:58.824014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:58.823959    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" podStartSLOduration=1.823945245 podStartE2EDuration="1.823945245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:58.822624494 +0000 UTC m=+2682.734864784" watchObservedRunningTime="2026-07-17 22:11:58.823945245 +0000 UTC m=+2682.736185534"
Jul 17 22:11:59.597354 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:59.597315    2563 scope.go:117] "RemoveContainer" containerID="4208de54cfc96b5f2d5cbc11c853cbd6b0c249d7683b3d9c98b4f2ef9d5e56a6"
Jul 17 22:11:59.814361 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:59.814328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" event={"ID":"6b3332b4-eedf-4151-8fad-91b114d6f907","Type":"ContainerStarted","Data":"4c716699cb7cf31cb856d255852ffcc9f7958b5faefe5f57c1ab14c856d23bd8"}
Jul 17 22:11:59.815867 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:59.815833    2563 generic.go:356] "Generic (PLEG): container finished" podID="dfe8f820-642e-45df-bd8d-6e770e976023" containerID="b99290aade22d02e8d79294fd83973ebf6ecb797ccc3217e4272da61f154e483" exitCode=6
Jul 17 22:11:59.815984 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:59.815868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerDied","Data":"b99290aade22d02e8d79294fd83973ebf6ecb797ccc3217e4272da61f154e483"}
Jul 17 22:11:59.816084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:59.816072    2563 scope.go:117] "RemoveContainer" containerID="b99290aade22d02e8d79294fd83973ebf6ecb797ccc3217e4272da61f154e483"
Jul 17 22:11:59.831568 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:11:59.831526    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" podStartSLOduration=16.831512678 podStartE2EDuration="16.831512678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:59.828300584 +0000 UTC m=+2683.740540872" watchObservedRunningTime="2026-07-17 22:11:59.831512678 +0000 UTC m=+2683.743752966"
Jul 17 22:12:00.823648 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.823539    2563 generic.go:356] "Generic (PLEG): container finished" podID="dfe8f820-642e-45df-bd8d-6e770e976023" containerID="ec4254893771f2ae89809cbafe79660351fe2d6f9c64728862bae73ab79b1f23" exitCode=6
Jul 17 22:12:00.823648 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.823634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerDied","Data":"ec4254893771f2ae89809cbafe79660351fe2d6f9c64728862bae73ab79b1f23"}
Jul 17 22:12:00.824166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.823704    2563 scope.go:117] "RemoveContainer" containerID="b99290aade22d02e8d79294fd83973ebf6ecb797ccc3217e4272da61f154e483"
Jul 17 22:12:00.824166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.823900    2563 scope.go:117] "RemoveContainer" containerID="ec4254893771f2ae89809cbafe79660351fe2d6f9c64728862bae73ab79b1f23"
Jul 17 22:12:00.824166 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:00.824134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw_odh-ai-gateway-infra(dfe8f820-642e-45df-bd8d-6e770e976023)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" podUID="dfe8f820-642e-45df-bd8d-6e770e976023"
Jul 17 22:12:00.825802 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.825780    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerID="4c716699cb7cf31cb856d255852ffcc9f7958b5faefe5f57c1ab14c856d23bd8" exitCode=6
Jul 17 22:12:00.825893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.825834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" event={"ID":"6b3332b4-eedf-4151-8fad-91b114d6f907","Type":"ContainerDied","Data":"4c716699cb7cf31cb856d255852ffcc9f7958b5faefe5f57c1ab14c856d23bd8"}
Jul 17 22:12:00.826098 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.826083    2563 scope.go:117] "RemoveContainer" containerID="4c716699cb7cf31cb856d255852ffcc9f7958b5faefe5f57c1ab14c856d23bd8"
Jul 17 22:12:00.826283 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:00.826268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-r64st_odh-ai-gateway-infra(6b3332b4-eedf-4151-8fad-91b114d6f907)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907"
Jul 17 22:12:00.835052 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.835029    2563 scope.go:117] "RemoveContainer" containerID="4208de54cfc96b5f2d5cbc11c853cbd6b0c249d7683b3d9c98b4f2ef9d5e56a6"
Jul 17 22:12:00.838455 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:00.838419    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"]
Jul 17 22:12:01.834425 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:01.834400    2563 scope.go:117] "RemoveContainer" containerID="ec4254893771f2ae89809cbafe79660351fe2d6f9c64728862bae73ab79b1f23"
Jul 17 22:12:01.834756 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:01.834696    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw_odh-ai-gateway-infra(dfe8f820-642e-45df-bd8d-6e770e976023)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" podUID="dfe8f820-642e-45df-bd8d-6e770e976023"
Jul 17 22:12:01.990133 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:01.990108    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:12:02.158787 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.158703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3332b4-eedf-4151-8fad-91b114d6f907-maas-api-service-ca\") pod \"6b3332b4-eedf-4151-8fad-91b114d6f907\" (UID: \"6b3332b4-eedf-4151-8fad-91b114d6f907\") "
Jul 17 22:12:02.159051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.159029    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3332b4-eedf-4151-8fad-91b114d6f907-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b3332b4-eedf-4151-8fad-91b114d6f907" (UID: "6b3332b4-eedf-4151-8fad-91b114d6f907"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:02.259558 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.259529    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3332b4-eedf-4151-8fad-91b114d6f907-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:12:02.840502 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.840475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"
Jul 17 22:12:02.840934 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.840475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st" event={"ID":"6b3332b4-eedf-4151-8fad-91b114d6f907","Type":"ContainerDied","Data":"b2ea3708e56fa69cb9bf08c71467270de88c78fc9a6630c59afd25bf463202f4"}
Jul 17 22:12:02.840934 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.840637    2563 scope.go:117] "RemoveContainer" containerID="4c716699cb7cf31cb856d255852ffcc9f7958b5faefe5f57c1ab14c856d23bd8"
Jul 17 22:12:02.855880 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.855856    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"]
Jul 17 22:12:02.861202 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:02.861152    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-r64st"]
Jul 17 22:12:04.602316 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:04.602282    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" path="/var/lib/kubelet/pods/6b3332b4-eedf-4151-8fad-91b114d6f907/volumes"
Jul 17 22:12:06.598983 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:06.598921    2563 scope.go:117] "RemoveContainer" containerID="a26d11ca429226a7ae2de12452a67a472d0d128c6adff804bf384318a24f0387"
Jul 17 22:12:06.862489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:06.862410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" event={"ID":"4a027125-34a9-41c6-810d-0e5831ba98d5","Type":"ContainerStarted","Data":"cc9f9d6536299014cabb7b2a0d093b34a792fb82bd018bd0d28eac65d00d2df9"}
Jul 17 22:12:06.878186 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:06.878138    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" podStartSLOduration=16.87812372 podStartE2EDuration="16.87812372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:12:06.876590598 +0000 UTC m=+2690.788830883" watchObservedRunningTime="2026-07-17 22:12:06.87812372 +0000 UTC m=+2690.790364009"
Jul 17 22:12:07.870059 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:07.869984    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerID="cc9f9d6536299014cabb7b2a0d093b34a792fb82bd018bd0d28eac65d00d2df9" exitCode=6
Jul 17 22:12:07.870059 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:07.870041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" event={"ID":"4a027125-34a9-41c6-810d-0e5831ba98d5","Type":"ContainerDied","Data":"cc9f9d6536299014cabb7b2a0d093b34a792fb82bd018bd0d28eac65d00d2df9"}
Jul 17 22:12:07.870524 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:07.870090    2563 scope.go:117] "RemoveContainer" containerID="a26d11ca429226a7ae2de12452a67a472d0d128c6adff804bf384318a24f0387"
Jul 17 22:12:07.870524 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:07.870377    2563 scope.go:117] "RemoveContainer" containerID="cc9f9d6536299014cabb7b2a0d093b34a792fb82bd018bd0d28eac65d00d2df9"
Jul 17 22:12:07.870662 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:07.870560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4_odh-ai-gateway-infra(4a027125-34a9-41c6-810d-0e5831ba98d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5"
Jul 17 22:12:07.886298 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:07.886278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"]
Jul 17 22:12:09.010353 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.010327    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:12:09.119912 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.119876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a027125-34a9-41c6-810d-0e5831ba98d5-maas-api-service-ca\") pod \"4a027125-34a9-41c6-810d-0e5831ba98d5\" (UID: \"4a027125-34a9-41c6-810d-0e5831ba98d5\") "
Jul 17 22:12:09.120309 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.120288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a027125-34a9-41c6-810d-0e5831ba98d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a027125-34a9-41c6-810d-0e5831ba98d5" (UID: "4a027125-34a9-41c6-810d-0e5831ba98d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:09.221023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.220976    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a027125-34a9-41c6-810d-0e5831ba98d5-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:12:09.883186 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.883157    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"
Jul 17 22:12:09.883387 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.883151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4" event={"ID":"4a027125-34a9-41c6-810d-0e5831ba98d5","Type":"ContainerDied","Data":"60332bc6732f9d99c5dcf12b19290f4ccb7c9dc7c2d3b679d5076a8916615dd6"}
Jul 17 22:12:09.883387 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.883306    2563 scope.go:117] "RemoveContainer" containerID="cc9f9d6536299014cabb7b2a0d093b34a792fb82bd018bd0d28eac65d00d2df9"
Jul 17 22:12:09.909133 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.909101    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"]
Jul 17 22:12:09.914690 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:09.914669    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-vzkc4"]
Jul 17 22:12:10.602632 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:10.602579    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" path="/var/lib/kubelet/pods/4a027125-34a9-41c6-810d-0e5831ba98d5/volumes"
Jul 17 22:12:15.597290 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:15.597254    2563 scope.go:117] "RemoveContainer" containerID="ec4254893771f2ae89809cbafe79660351fe2d6f9c64728862bae73ab79b1f23"
Jul 17 22:12:15.916767 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:15.916736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerStarted","Data":"bf94ecb2a31fd5ec412657f25984714e664eba62932c74fa23309c0b382376b0"}
Jul 17 22:12:16.752802 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.752760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:12:16.759993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.759971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:12:16.924268 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.924224    2563 generic.go:356] "Generic (PLEG): container finished" podID="dfe8f820-642e-45df-bd8d-6e770e976023" containerID="bf94ecb2a31fd5ec412657f25984714e664eba62932c74fa23309c0b382376b0" exitCode=6
Jul 17 22:12:16.924450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.924274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerDied","Data":"bf94ecb2a31fd5ec412657f25984714e664eba62932c74fa23309c0b382376b0"}
Jul 17 22:12:16.924450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.924341    2563 scope.go:117] "RemoveContainer" containerID="ec4254893771f2ae89809cbafe79660351fe2d6f9c64728862bae73ab79b1f23"
Jul 17 22:12:16.924682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.924665    2563 scope.go:117] "RemoveContainer" containerID="bf94ecb2a31fd5ec412657f25984714e664eba62932c74fa23309c0b382376b0"
Jul 17 22:12:16.924882 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:16.924859    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw_odh-ai-gateway-infra(dfe8f820-642e-45df-bd8d-6e770e976023)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" podUID="dfe8f820-642e-45df-bd8d-6e770e976023"
Jul 17 22:12:16.940425 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:16.940403    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"]
Jul 17 22:12:18.067573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.067544    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:12:18.088263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.088206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f820-642e-45df-bd8d-6e770e976023-maas-api-service-ca\") pod \"dfe8f820-642e-45df-bd8d-6e770e976023\" (UID: \"dfe8f820-642e-45df-bd8d-6e770e976023\") "
Jul 17 22:12:18.088681 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.088634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe8f820-642e-45df-bd8d-6e770e976023-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfe8f820-642e-45df-bd8d-6e770e976023" (UID: "dfe8f820-642e-45df-bd8d-6e770e976023"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:18.189060 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.188978    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f820-642e-45df-bd8d-6e770e976023-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:12:18.937460 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.937431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"
Jul 17 22:12:18.937678 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.937475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw" event={"ID":"dfe8f820-642e-45df-bd8d-6e770e976023","Type":"ContainerDied","Data":"a976d6310fb656b7dfc1447d0fe2a0544e98302faa29a01010f42a443c6f20cc"}
Jul 17 22:12:18.937678 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.937536    2563 scope.go:117] "RemoveContainer" containerID="bf94ecb2a31fd5ec412657f25984714e664eba62932c74fa23309c0b382376b0"
Jul 17 22:12:18.952201 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.952178    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"]
Jul 17 22:12:18.957684 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:18.957660    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-rfdjw"]
Jul 17 22:12:20.603298 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:20.603263    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" path="/var/lib/kubelet/pods/dfe8f820-642e-45df-bd8d-6e770e976023/volumes"
Jul 17 22:12:25.751360 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"]
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751710    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751715    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751729    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751736    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751753    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751758    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751765    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.751764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751778    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751783    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751842    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751850    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751856    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751862    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751868    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751872    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751878    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:25.752153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.751885    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8ef13c-1ecc-4458-9591-044e53e6e28a" containerName="revoke-keys"
Jul 17 22:12:25.754562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.754546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:25.757003 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.756983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:12:25.760939 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.760912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"]
Jul 17 22:12:25.843044 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.843005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11eb276a-db2b-4109-8036-844a1f849e4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr\" (UID: \"11eb276a-db2b-4109-8036-844a1f849e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:25.944312 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.944269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11eb276a-db2b-4109-8036-844a1f849e4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr\" (UID: \"11eb276a-db2b-4109-8036-844a1f849e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:25.944936 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:25.944916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11eb276a-db2b-4109-8036-844a1f849e4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr\" (UID: \"11eb276a-db2b-4109-8036-844a1f849e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:26.066229 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:26.066139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:26.213890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:26.205466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"]
Jul 17 22:12:26.979848 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:26.979805    2563 generic.go:356] "Generic (PLEG): container finished" podID="11eb276a-db2b-4109-8036-844a1f849e4f" containerID="dd938ccb57a36a5987017dcb912d365d4ab7fe4c82c1888347fdd9e1a79a41cc" exitCode=6
Jul 17 22:12:26.980220 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:26.979892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" event={"ID":"11eb276a-db2b-4109-8036-844a1f849e4f","Type":"ContainerDied","Data":"dd938ccb57a36a5987017dcb912d365d4ab7fe4c82c1888347fdd9e1a79a41cc"}
Jul 17 22:12:26.980220 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:26.979944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" event={"ID":"11eb276a-db2b-4109-8036-844a1f849e4f","Type":"ContainerStarted","Data":"c9f46acacea555b873bde840f82b4615e280cc162ed7c802dd0b6863f0d2ecb4"}
Jul 17 22:12:26.980220 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:26.980050    2563 scope.go:117] "RemoveContainer" containerID="dd938ccb57a36a5987017dcb912d365d4ab7fe4c82c1888347fdd9e1a79a41cc"
Jul 17 22:12:27.986236 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:27.986196    2563 generic.go:356] "Generic (PLEG): container finished" podID="11eb276a-db2b-4109-8036-844a1f849e4f" containerID="53d962096b6d44106c8eba0e15053f9ed2a3988f5d83bbdbc1ed8cd1faac29a4" exitCode=6
Jul 17 22:12:27.986740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:27.986285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" event={"ID":"11eb276a-db2b-4109-8036-844a1f849e4f","Type":"ContainerDied","Data":"53d962096b6d44106c8eba0e15053f9ed2a3988f5d83bbdbc1ed8cd1faac29a4"}
Jul 17 22:12:27.986740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:27.986357    2563 scope.go:117] "RemoveContainer" containerID="dd938ccb57a36a5987017dcb912d365d4ab7fe4c82c1888347fdd9e1a79a41cc"
Jul 17 22:12:27.986740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:27.986556    2563 scope.go:117] "RemoveContainer" containerID="53d962096b6d44106c8eba0e15053f9ed2a3988f5d83bbdbc1ed8cd1faac29a4"
Jul 17 22:12:27.986972 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:27.986797    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr_odh-ai-gateway-infra(11eb276a-db2b-4109-8036-844a1f849e4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" podUID="11eb276a-db2b-4109-8036-844a1f849e4f"
Jul 17 22:12:28.992161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:28.992136    2563 scope.go:117] "RemoveContainer" containerID="53d962096b6d44106c8eba0e15053f9ed2a3988f5d83bbdbc1ed8cd1faac29a4"
Jul 17 22:12:28.992556 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:28.992346    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr_odh-ai-gateway-infra(11eb276a-db2b-4109-8036-844a1f849e4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" podUID="11eb276a-db2b-4109-8036-844a1f849e4f"
Jul 17 22:12:33.217379 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.217273    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"]
Jul 17 22:12:33.219669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.218147    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:33.219669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.218172    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:33.219669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.218193    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:33.219669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.218202    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3332b4-eedf-4151-8fad-91b114d6f907" containerName="revoke-keys"
Jul 17 22:12:33.219669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.218410    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a027125-34a9-41c6-810d-0e5831ba98d5" containerName="revoke-keys"
Jul 17 22:12:33.219669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.218423    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe8f820-642e-45df-bd8d-6e770e976023" containerName="revoke-keys"
Jul 17 22:12:33.222797 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.222770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:33.224091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.224056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"]
Jul 17 22:12:33.306341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.306307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg\" (UID: \"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:33.407156 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.407121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg\" (UID: \"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:33.407811 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.407792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg\" (UID: \"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:33.535125 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.535034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:33.674262 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:33.674239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"]
Jul 17 22:12:33.676955 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:12:33.676925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ba0a8f_3d5b_420f_b1c6_6e929f12c6c7.slice/crio-0ea48c4ea50162a0517c113b5268e6a1d9a76607b185543698c39b23b679136f WatchSource:0}: Error finding container 0ea48c4ea50162a0517c113b5268e6a1d9a76607b185543698c39b23b679136f: Status 404 returned error can't find the container with id 0ea48c4ea50162a0517c113b5268e6a1d9a76607b185543698c39b23b679136f
Jul 17 22:12:34.017639 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:34.017590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerStarted","Data":"6c52d6467405d5acda1acf8b97362e438a021ce365fa0e442aa21ec0afda41ec"}
Jul 17 22:12:34.017789 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:34.017657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerStarted","Data":"0ea48c4ea50162a0517c113b5268e6a1d9a76607b185543698c39b23b679136f"}
Jul 17 22:12:34.032419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:34.032371    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" podStartSLOduration=1.032356398 podStartE2EDuration="1.032356398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:12:34.031613619 +0000 UTC m=+2717.943853900" watchObservedRunningTime="2026-07-17 22:12:34.032356398 +0000 UTC m=+2717.944596687"
Jul 17 22:12:35.023873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:35.023837    2563 generic.go:356] "Generic (PLEG): container finished" podID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerID="6c52d6467405d5acda1acf8b97362e438a021ce365fa0e442aa21ec0afda41ec" exitCode=6
Jul 17 22:12:35.024289 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:35.023918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerDied","Data":"6c52d6467405d5acda1acf8b97362e438a021ce365fa0e442aa21ec0afda41ec"}
Jul 17 22:12:35.024289 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:35.024069    2563 scope.go:117] "RemoveContainer" containerID="6c52d6467405d5acda1acf8b97362e438a021ce365fa0e442aa21ec0afda41ec"
Jul 17 22:12:36.030794 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:36.030761    2563 generic.go:356] "Generic (PLEG): container finished" podID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerID="19d422f90d89ceed7ec17fdf3d3e634037a1a1d9bbb722a97d511d817a4ca86a" exitCode=6
Jul 17 22:12:36.031292 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:36.030851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerDied","Data":"19d422f90d89ceed7ec17fdf3d3e634037a1a1d9bbb722a97d511d817a4ca86a"}
Jul 17 22:12:36.031292 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:36.030912    2563 scope.go:117] "RemoveContainer" containerID="6c52d6467405d5acda1acf8b97362e438a021ce365fa0e442aa21ec0afda41ec"
Jul 17 22:12:36.031292 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:36.031130    2563 scope.go:117] "RemoveContainer" containerID="19d422f90d89ceed7ec17fdf3d3e634037a1a1d9bbb722a97d511d817a4ca86a"
Jul 17 22:12:36.031415 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:36.031339    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg_odh-ai-gateway-infra(48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7"
Jul 17 22:12:37.037083 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:37.037057    2563 scope.go:117] "RemoveContainer" containerID="19d422f90d89ceed7ec17fdf3d3e634037a1a1d9bbb722a97d511d817a4ca86a"
Jul 17 22:12:37.037542 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:37.037233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg_odh-ai-gateway-infra(48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7"
Jul 17 22:12:39.597104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:39.597072    2563 scope.go:117] "RemoveContainer" containerID="53d962096b6d44106c8eba0e15053f9ed2a3988f5d83bbdbc1ed8cd1faac29a4"
Jul 17 22:12:40.053525 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.053481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" event={"ID":"11eb276a-db2b-4109-8036-844a1f849e4f","Type":"ContainerStarted","Data":"05704a1a03258faa832376a713242a5de94bf0d785b2c779a255468879b35d4a"}
Jul 17 22:12:40.061500 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.061472    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"]
Jul 17 22:12:40.064837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.064819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:40.068727 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.068689    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" podStartSLOduration=15.068677552 podStartE2EDuration="15.068677552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:12:40.068434 +0000 UTC m=+2723.980674290" watchObservedRunningTime="2026-07-17 22:12:40.068677552 +0000 UTC m=+2723.980917840"
Jul 17 22:12:40.074189 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.074156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"]
Jul 17 22:12:40.166258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.166220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7802-d088-4375-917a-62ab4d2666ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h\" (UID: \"eafc7802-d088-4375-917a-62ab4d2666ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:40.266975 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.266941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7802-d088-4375-917a-62ab4d2666ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h\" (UID: \"eafc7802-d088-4375-917a-62ab4d2666ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:40.267504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.267487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7802-d088-4375-917a-62ab4d2666ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h\" (UID: \"eafc7802-d088-4375-917a-62ab4d2666ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:40.376435 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.376349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:40.511460 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:40.511433    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"]
Jul 17 22:12:40.513247 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:12:40.513218    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafc7802_d088_4375_917a_62ab4d2666ab.slice/crio-e05ea496f82fe179223cd750c8eaad665a43037dc184dbd948b1aefeb7bd75f5 WatchSource:0}: Error finding container e05ea496f82fe179223cd750c8eaad665a43037dc184dbd948b1aefeb7bd75f5: Status 404 returned error can't find the container with id e05ea496f82fe179223cd750c8eaad665a43037dc184dbd948b1aefeb7bd75f5
Jul 17 22:12:41.059991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.059893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerStarted","Data":"0ff23fbb65d0350d75da437c8ff018cd5a9a22de98c441e1662c1fe8cb921bcb"}
Jul 17 22:12:41.059991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.059953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerStarted","Data":"e05ea496f82fe179223cd750c8eaad665a43037dc184dbd948b1aefeb7bd75f5"}
Jul 17 22:12:41.061564 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.061537    2563 generic.go:356] "Generic (PLEG): container finished" podID="11eb276a-db2b-4109-8036-844a1f849e4f" containerID="05704a1a03258faa832376a713242a5de94bf0d785b2c779a255468879b35d4a" exitCode=6
Jul 17 22:12:41.061698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.061620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" event={"ID":"11eb276a-db2b-4109-8036-844a1f849e4f","Type":"ContainerDied","Data":"05704a1a03258faa832376a713242a5de94bf0d785b2c779a255468879b35d4a"}
Jul 17 22:12:41.061698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.061668    2563 scope.go:117] "RemoveContainer" containerID="53d962096b6d44106c8eba0e15053f9ed2a3988f5d83bbdbc1ed8cd1faac29a4"
Jul 17 22:12:41.061927 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.061904    2563 scope.go:117] "RemoveContainer" containerID="05704a1a03258faa832376a713242a5de94bf0d785b2c779a255468879b35d4a"
Jul 17 22:12:41.062121 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:41.062101    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr_odh-ai-gateway-infra(11eb276a-db2b-4109-8036-844a1f849e4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" podUID="11eb276a-db2b-4109-8036-844a1f849e4f"
Jul 17 22:12:41.073378 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.073333    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" podStartSLOduration=1.073315932 podStartE2EDuration="1.073315932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:12:41.073296958 +0000 UTC m=+2724.985537248" watchObservedRunningTime="2026-07-17 22:12:41.073315932 +0000 UTC m=+2724.985556221"
Jul 17 22:12:41.079732 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:41.079708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"]
Jul 17 22:12:42.068497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.068469    2563 generic.go:356] "Generic (PLEG): container finished" podID="eafc7802-d088-4375-917a-62ab4d2666ab" containerID="0ff23fbb65d0350d75da437c8ff018cd5a9a22de98c441e1662c1fe8cb921bcb" exitCode=6
Jul 17 22:12:42.068975 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.068561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerDied","Data":"0ff23fbb65d0350d75da437c8ff018cd5a9a22de98c441e1662c1fe8cb921bcb"}
Jul 17 22:12:42.068975 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.068749    2563 scope.go:117] "RemoveContainer" containerID="0ff23fbb65d0350d75da437c8ff018cd5a9a22de98c441e1662c1fe8cb921bcb"
Jul 17 22:12:42.216033 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.216003    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:42.283462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.283435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11eb276a-db2b-4109-8036-844a1f849e4f-maas-api-service-ca\") pod \"11eb276a-db2b-4109-8036-844a1f849e4f\" (UID: \"11eb276a-db2b-4109-8036-844a1f849e4f\") "
Jul 17 22:12:42.283788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.283765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11eb276a-db2b-4109-8036-844a1f849e4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11eb276a-db2b-4109-8036-844a1f849e4f" (UID: "11eb276a-db2b-4109-8036-844a1f849e4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:42.384449 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:42.384427    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11eb276a-db2b-4109-8036-844a1f849e4f-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:12:43.076723 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.076690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"
Jul 17 22:12:43.077151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.076720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr" event={"ID":"11eb276a-db2b-4109-8036-844a1f849e4f","Type":"ContainerDied","Data":"c9f46acacea555b873bde840f82b4615e280cc162ed7c802dd0b6863f0d2ecb4"}
Jul 17 22:12:43.077151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.076782    2563 scope.go:117] "RemoveContainer" containerID="05704a1a03258faa832376a713242a5de94bf0d785b2c779a255468879b35d4a"
Jul 17 22:12:43.078835 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.078805    2563 generic.go:356] "Generic (PLEG): container finished" podID="eafc7802-d088-4375-917a-62ab4d2666ab" containerID="5bf95e61e6a9f1ceec40fa6d950c3144b18bda831c5be60932cb9db79d5cfacb" exitCode=6
Jul 17 22:12:43.078957 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.078876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerDied","Data":"5bf95e61e6a9f1ceec40fa6d950c3144b18bda831c5be60932cb9db79d5cfacb"}
Jul 17 22:12:43.079125 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.079104    2563 scope.go:117] "RemoveContainer" containerID="5bf95e61e6a9f1ceec40fa6d950c3144b18bda831c5be60932cb9db79d5cfacb"
Jul 17 22:12:43.079374 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:43.079353    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h_odh-ai-gateway-infra(eafc7802-d088-4375-917a-62ab4d2666ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" podUID="eafc7802-d088-4375-917a-62ab4d2666ab"
Jul 17 22:12:43.085546 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.085530    2563 scope.go:117] "RemoveContainer" containerID="0ff23fbb65d0350d75da437c8ff018cd5a9a22de98c441e1662c1fe8cb921bcb"
Jul 17 22:12:43.105933 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.105900    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"]
Jul 17 22:12:43.113121 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:43.113099    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-t6crr"]
Jul 17 22:12:44.085530 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:44.085504    2563 scope.go:117] "RemoveContainer" containerID="5bf95e61e6a9f1ceec40fa6d950c3144b18bda831c5be60932cb9db79d5cfacb"
Jul 17 22:12:44.085999 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:44.085721    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h_odh-ai-gateway-infra(eafc7802-d088-4375-917a-62ab4d2666ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" podUID="eafc7802-d088-4375-917a-62ab4d2666ab"
Jul 17 22:12:44.603386 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:44.603348    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" path="/var/lib/kubelet/pods/11eb276a-db2b-4109-8036-844a1f849e4f/volumes"
Jul 17 22:12:51.596854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:51.596825    2563 scope.go:117] "RemoveContainer" containerID="19d422f90d89ceed7ec17fdf3d3e634037a1a1d9bbb722a97d511d817a4ca86a"
Jul 17 22:12:52.126404 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:52.126309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerStarted","Data":"7251c2baaba9dd134cfe0d611c5593df70b48ce71d246d83e7aac251dcd2cda7"}
Jul 17 22:12:53.133039 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:53.133005    2563 generic.go:356] "Generic (PLEG): container finished" podID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerID="7251c2baaba9dd134cfe0d611c5593df70b48ce71d246d83e7aac251dcd2cda7" exitCode=6
Jul 17 22:12:53.133441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:53.133076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerDied","Data":"7251c2baaba9dd134cfe0d611c5593df70b48ce71d246d83e7aac251dcd2cda7"}
Jul 17 22:12:53.133441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:53.133135    2563 scope.go:117] "RemoveContainer" containerID="19d422f90d89ceed7ec17fdf3d3e634037a1a1d9bbb722a97d511d817a4ca86a"
Jul 17 22:12:53.133571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:53.133495    2563 scope.go:117] "RemoveContainer" containerID="7251c2baaba9dd134cfe0d611c5593df70b48ce71d246d83e7aac251dcd2cda7"
Jul 17 22:12:53.133741 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:53.133722    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg_odh-ai-gateway-infra(48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7"
Jul 17 22:12:53.158531 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:53.158497    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"]
Jul 17 22:12:54.272456 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:54.272431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:54.386824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:54.386793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7-maas-api-service-ca\") pod \"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7\" (UID: \"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7\") "
Jul 17 22:12:54.387182 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:54.387156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" (UID: "48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:54.488075 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:54.488047    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:12:54.596662 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:54.596632    2563 scope.go:117] "RemoveContainer" containerID="5bf95e61e6a9f1ceec40fa6d950c3144b18bda831c5be60932cb9db79d5cfacb"
Jul 17 22:12:55.145976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:55.145885    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"
Jul 17 22:12:55.145976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:55.145914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg" event={"ID":"48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7","Type":"ContainerDied","Data":"0ea48c4ea50162a0517c113b5268e6a1d9a76607b185543698c39b23b679136f"}
Jul 17 22:12:55.145976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:55.145972    2563 scope.go:117] "RemoveContainer" containerID="7251c2baaba9dd134cfe0d611c5593df70b48ce71d246d83e7aac251dcd2cda7"
Jul 17 22:12:55.147955 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:55.147916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerStarted","Data":"4f635dff6f209a118f40910e5ec37bdde8a4a5b49bb36c45a1572216cb865374"}
Jul 17 22:12:55.161072 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:55.161050    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"]
Jul 17 22:12:55.173505 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:55.173476    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jwlkg"]
Jul 17 22:12:56.155034 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:56.155001    2563 generic.go:356] "Generic (PLEG): container finished" podID="eafc7802-d088-4375-917a-62ab4d2666ab" containerID="4f635dff6f209a118f40910e5ec37bdde8a4a5b49bb36c45a1572216cb865374" exitCode=6
Jul 17 22:12:56.155499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:56.155077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerDied","Data":"4f635dff6f209a118f40910e5ec37bdde8a4a5b49bb36c45a1572216cb865374"}
Jul 17 22:12:56.155499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:56.155140    2563 scope.go:117] "RemoveContainer" containerID="5bf95e61e6a9f1ceec40fa6d950c3144b18bda831c5be60932cb9db79d5cfacb"
Jul 17 22:12:56.155499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:56.155440    2563 scope.go:117] "RemoveContainer" containerID="4f635dff6f209a118f40910e5ec37bdde8a4a5b49bb36c45a1572216cb865374"
Jul 17 22:12:56.155722 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:12:56.155708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h_odh-ai-gateway-infra(eafc7802-d088-4375-917a-62ab4d2666ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" podUID="eafc7802-d088-4375-917a-62ab4d2666ab"
Jul 17 22:12:56.195853 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:56.195815    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"]
Jul 17 22:12:56.603345 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:56.603309    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" path="/var/lib/kubelet/pods/48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7/volumes"
Jul 17 22:12:57.309505 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.309484    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:57.414924 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.414842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7802-d088-4375-917a-62ab4d2666ab-maas-api-service-ca\") pod \"eafc7802-d088-4375-917a-62ab4d2666ab\" (UID: \"eafc7802-d088-4375-917a-62ab4d2666ab\") "
Jul 17 22:12:57.415172 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.415151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafc7802-d088-4375-917a-62ab4d2666ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eafc7802-d088-4375-917a-62ab4d2666ab" (UID: "eafc7802-d088-4375-917a-62ab4d2666ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:57.516273 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.516231    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7802-d088-4375-917a-62ab4d2666ab-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:12:57.876270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876185    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"]
Jul 17 22:12:57.876590 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:12:57.876590 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876618    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876630    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876636    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876648    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876668    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876683    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:12:57.876702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876687    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:12:57.877045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876743    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:12:57.877045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876750    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:12:57.877045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876758    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:12:57.877045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876764    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:12:57.877045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876771    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:12:57.877045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.876777    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:12:57.879660 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.879642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:12:57.887399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:57.887165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"]
Jul 17 22:12:58.021002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.020958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb865c6c-5e29-4d29-ac7f-7819f3da430a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj\" (UID: \"fb865c6c-5e29-4d29-ac7f-7819f3da430a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:12:58.122589 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.122549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb865c6c-5e29-4d29-ac7f-7819f3da430a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj\" (UID: \"fb865c6c-5e29-4d29-ac7f-7819f3da430a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:12:58.123168 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.123148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb865c6c-5e29-4d29-ac7f-7819f3da430a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj\" (UID: \"fb865c6c-5e29-4d29-ac7f-7819f3da430a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:12:58.169381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.169346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h" event={"ID":"eafc7802-d088-4375-917a-62ab4d2666ab","Type":"ContainerDied","Data":"e05ea496f82fe179223cd750c8eaad665a43037dc184dbd948b1aefeb7bd75f5"}
Jul 17 22:12:58.169381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.169376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"
Jul 17 22:12:58.169633 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.169416    2563 scope.go:117] "RemoveContainer" containerID="4f635dff6f209a118f40910e5ec37bdde8a4a5b49bb36c45a1572216cb865374"
Jul 17 22:12:58.191375 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.191352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:12:58.196467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.196446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"]
Jul 17 22:12:58.201037 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.201015    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-8hc4h"]
Jul 17 22:12:58.330634 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.330583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"]
Jul 17 22:12:58.332672 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:12:58.332647    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb865c6c_5e29_4d29_ac7f_7819f3da430a.slice/crio-94caff46085c3d4e2198a5053ee737801fae7dbd2f0caa6173a88829e1d1aec4 WatchSource:0}: Error finding container 94caff46085c3d4e2198a5053ee737801fae7dbd2f0caa6173a88829e1d1aec4: Status 404 returned error can't find the container with id 94caff46085c3d4e2198a5053ee737801fae7dbd2f0caa6173a88829e1d1aec4
Jul 17 22:12:58.602721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:58.602692    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" path="/var/lib/kubelet/pods/eafc7802-d088-4375-917a-62ab4d2666ab/volumes"
Jul 17 22:12:59.176584 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:59.176547    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerID="32e807340f6ba343557ed8eadd72f8b47138111554f62b045cc2aa70d02df39b" exitCode=6
Jul 17 22:12:59.176747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:59.176630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" event={"ID":"fb865c6c-5e29-4d29-ac7f-7819f3da430a","Type":"ContainerDied","Data":"32e807340f6ba343557ed8eadd72f8b47138111554f62b045cc2aa70d02df39b"}
Jul 17 22:12:59.176747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:59.176685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" event={"ID":"fb865c6c-5e29-4d29-ac7f-7819f3da430a","Type":"ContainerStarted","Data":"94caff46085c3d4e2198a5053ee737801fae7dbd2f0caa6173a88829e1d1aec4"}
Jul 17 22:12:59.176940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:12:59.176927    2563 scope.go:117] "RemoveContainer" containerID="32e807340f6ba343557ed8eadd72f8b47138111554f62b045cc2aa70d02df39b"
Jul 17 22:13:00.183183 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:00.183144    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerID="e777f85c936a926b9931296eef414362dfff33aef6d77f945606c1afccbb1285" exitCode=6
Jul 17 22:13:00.183652 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:00.183230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" event={"ID":"fb865c6c-5e29-4d29-ac7f-7819f3da430a","Type":"ContainerDied","Data":"e777f85c936a926b9931296eef414362dfff33aef6d77f945606c1afccbb1285"}
Jul 17 22:13:00.183652 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:00.183336    2563 scope.go:117] "RemoveContainer" containerID="32e807340f6ba343557ed8eadd72f8b47138111554f62b045cc2aa70d02df39b"
Jul 17 22:13:00.183652 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:00.183467    2563 scope.go:117] "RemoveContainer" containerID="e777f85c936a926b9931296eef414362dfff33aef6d77f945606c1afccbb1285"
Jul 17 22:13:00.183844 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:13:00.183737    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj_odh-ai-gateway-infra(fb865c6c-5e29-4d29-ac7f-7819f3da430a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a"
Jul 17 22:13:01.190226 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:01.190197    2563 scope.go:117] "RemoveContainer" containerID="e777f85c936a926b9931296eef414362dfff33aef6d77f945606c1afccbb1285"
Jul 17 22:13:01.190613 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:13:01.190394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj_odh-ai-gateway-infra(fb865c6c-5e29-4d29-ac7f-7819f3da430a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a"
Jul 17 22:13:12.597498 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:12.597458    2563 scope.go:117] "RemoveContainer" containerID="e777f85c936a926b9931296eef414362dfff33aef6d77f945606c1afccbb1285"
Jul 17 22:13:13.251286 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:13.251253    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerID="9c480083c48fe82a26cd663aa7ac4f20c2adaf04cf242ed18cc105d3b7fe8877" exitCode=6
Jul 17 22:13:13.251457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:13.251321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" event={"ID":"fb865c6c-5e29-4d29-ac7f-7819f3da430a","Type":"ContainerDied","Data":"9c480083c48fe82a26cd663aa7ac4f20c2adaf04cf242ed18cc105d3b7fe8877"}
Jul 17 22:13:13.251457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:13.251379    2563 scope.go:117] "RemoveContainer" containerID="e777f85c936a926b9931296eef414362dfff33aef6d77f945606c1afccbb1285"
Jul 17 22:13:13.251778 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:13.251754    2563 scope.go:117] "RemoveContainer" containerID="9c480083c48fe82a26cd663aa7ac4f20c2adaf04cf242ed18cc105d3b7fe8877"
Jul 17 22:13:13.251999 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:13:13.251980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj_odh-ai-gateway-infra(fb865c6c-5e29-4d29-ac7f-7819f3da430a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a"
Jul 17 22:13:14.276448 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:14.276418    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"]
Jul 17 22:13:14.414906 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:14.414884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:13:14.565321 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:14.565223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb865c6c-5e29-4d29-ac7f-7819f3da430a-maas-api-service-ca\") pod \"fb865c6c-5e29-4d29-ac7f-7819f3da430a\" (UID: \"fb865c6c-5e29-4d29-ac7f-7819f3da430a\") "
Jul 17 22:13:14.565615 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:14.565568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb865c6c-5e29-4d29-ac7f-7819f3da430a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb865c6c-5e29-4d29-ac7f-7819f3da430a" (UID: "fb865c6c-5e29-4d29-ac7f-7819f3da430a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:14.666609 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:14.666572    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb865c6c-5e29-4d29-ac7f-7819f3da430a-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:13:15.264187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:15.264148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj" event={"ID":"fb865c6c-5e29-4d29-ac7f-7819f3da430a","Type":"ContainerDied","Data":"94caff46085c3d4e2198a5053ee737801fae7dbd2f0caa6173a88829e1d1aec4"}
Jul 17 22:13:15.264390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:15.264217    2563 scope.go:117] "RemoveContainer" containerID="9c480083c48fe82a26cd663aa7ac4f20c2adaf04cf242ed18cc105d3b7fe8877"
Jul 17 22:13:15.264390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:15.264163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"
Jul 17 22:13:15.281070 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:15.281044    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"]
Jul 17 22:13:15.288025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:15.288003    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5lwzj"]
Jul 17 22:13:16.602574 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:16.602544    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" path="/var/lib/kubelet/pods/fb865c6c-5e29-4d29-ac7f-7819f3da430a/volumes"
Jul 17 22:13:47.914185 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914101    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"]
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914454    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914466    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914480    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914485    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914500    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914518    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914528    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914534    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914587    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914608    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11eb276a-db2b-4109-8036-844a1f849e4f" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914616    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafc7802-d088-4375-917a-62ab4d2666ab" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914622    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:13:47.914674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.914628    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ba0a8f-3d5b-420f-b1c6-6e929f12c6c7" containerName="revoke-keys"
Jul 17 22:13:47.918166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.918136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:13:47.921190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.921169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:13:47.928037 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:47.928014    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"]
Jul 17 22:13:48.044489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:48.044452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs\" (UID: \"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:13:48.145319 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:48.145281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs\" (UID: \"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:13:48.146034 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:48.146014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs\" (UID: \"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:13:48.231101 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:48.231021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:13:48.372753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:48.372725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"]
Jul 17 22:13:48.374930 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:13:48.374907    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5e2545_a4ad_4fe7_9b80_8605a7c813ec.slice/crio-6fd90676435d2a5c451657cd56e9a6d58d1472cb42c4f7129b9ea6200aefcb99 WatchSource:0}: Error finding container 6fd90676435d2a5c451657cd56e9a6d58d1472cb42c4f7129b9ea6200aefcb99: Status 404 returned error can't find the container with id 6fd90676435d2a5c451657cd56e9a6d58d1472cb42c4f7129b9ea6200aefcb99
Jul 17 22:13:48.429861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:48.429834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" event={"ID":"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec","Type":"ContainerStarted","Data":"6fd90676435d2a5c451657cd56e9a6d58d1472cb42c4f7129b9ea6200aefcb99"}
Jul 17 22:13:49.436103 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:49.436073    2563 generic.go:356] "Generic (PLEG): container finished" podID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerID="a831e40b12399c9e167e3a979ebb7df0adbbe8ed1773b5ed2dcaed60c39d7285" exitCode=6
Jul 17 22:13:49.436510 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:49.436159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" event={"ID":"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec","Type":"ContainerDied","Data":"a831e40b12399c9e167e3a979ebb7df0adbbe8ed1773b5ed2dcaed60c39d7285"}
Jul 17 22:13:49.436510 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:49.436323    2563 scope.go:117] "RemoveContainer" containerID="a831e40b12399c9e167e3a979ebb7df0adbbe8ed1773b5ed2dcaed60c39d7285"
Jul 17 22:13:50.442642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:50.442589    2563 generic.go:356] "Generic (PLEG): container finished" podID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerID="aef23cfc43fef62690a863e667b4105650e5ab91021f31d39de2bc62bf633d58" exitCode=6
Jul 17 22:13:50.442642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:50.442633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" event={"ID":"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec","Type":"ContainerDied","Data":"aef23cfc43fef62690a863e667b4105650e5ab91021f31d39de2bc62bf633d58"}
Jul 17 22:13:50.443178 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:50.442736    2563 scope.go:117] "RemoveContainer" containerID="a831e40b12399c9e167e3a979ebb7df0adbbe8ed1773b5ed2dcaed60c39d7285"
Jul 17 22:13:50.443178 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:50.442933    2563 scope.go:117] "RemoveContainer" containerID="aef23cfc43fef62690a863e667b4105650e5ab91021f31d39de2bc62bf633d58"
Jul 17 22:13:50.443272 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:13:50.443176    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs_odh-ai-gateway-infra(1f5e2545-a4ad-4fe7-9b80-8605a7c813ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec"
Jul 17 22:13:51.449380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:13:51.449348    2563 scope.go:117] "RemoveContainer" containerID="aef23cfc43fef62690a863e667b4105650e5ab91021f31d39de2bc62bf633d58"
Jul 17 22:13:51.449825 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:13:51.449526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs_odh-ai-gateway-infra(1f5e2545-a4ad-4fe7-9b80-8605a7c813ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec"
Jul 17 22:14:05.826499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.826463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"]
Jul 17 22:14:05.826922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.826844    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:14:05.826922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.826856    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:14:05.826922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.826922    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb865c6c-5e29-4d29-ac7f-7819f3da430a" containerName="revoke-keys"
Jul 17 22:14:05.829950 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.829934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:05.836788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.836763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"]
Jul 17 22:14:05.897848 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.897812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e438c9-70be-435b-8cc6-30bbbee13d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb\" (UID: \"c7e438c9-70be-435b-8cc6-30bbbee13d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:05.998930 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.998893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e438c9-70be-435b-8cc6-30bbbee13d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb\" (UID: \"c7e438c9-70be-435b-8cc6-30bbbee13d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:05.999513 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:05.999491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e438c9-70be-435b-8cc6-30bbbee13d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb\" (UID: \"c7e438c9-70be-435b-8cc6-30bbbee13d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:06.141101 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:06.141007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:06.278205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:06.278183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"]
Jul 17 22:14:06.280588 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:06.280564    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e438c9_70be_435b_8cc6_30bbbee13d6d.slice/crio-ac27d092155d3d4b12cc9b519b8937a6abf298487707382c481ea8de758f6a6f WatchSource:0}: Error finding container ac27d092155d3d4b12cc9b519b8937a6abf298487707382c481ea8de758f6a6f: Status 404 returned error can't find the container with id ac27d092155d3d4b12cc9b519b8937a6abf298487707382c481ea8de758f6a6f
Jul 17 22:14:06.521272 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:06.521238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" event={"ID":"c7e438c9-70be-435b-8cc6-30bbbee13d6d","Type":"ContainerStarted","Data":"b5bd4de4e1fc0cfdeb7d3a9fcd43d56b20dc351bcffdf9e2a4c06fe1b9aa4613"}
Jul 17 22:14:06.521403 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:06.521288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" event={"ID":"c7e438c9-70be-435b-8cc6-30bbbee13d6d","Type":"ContainerStarted","Data":"ac27d092155d3d4b12cc9b519b8937a6abf298487707382c481ea8de758f6a6f"}
Jul 17 22:14:06.536441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:06.536395    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" podStartSLOduration=1.5363777060000001 podStartE2EDuration="1.536377706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:06.535853269 +0000 UTC m=+2810.448093572" watchObservedRunningTime="2026-07-17 22:14:06.536377706 +0000 UTC m=+2810.448617997"
Jul 17 22:14:06.599833 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:06.599789    2563 scope.go:117] "RemoveContainer" containerID="aef23cfc43fef62690a863e667b4105650e5ab91021f31d39de2bc62bf633d58"
Jul 17 22:14:07.528125 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.528093    2563 generic.go:356] "Generic (PLEG): container finished" podID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerID="d6efab2e2eb29c3a998b3d8d6d85535c46955d4d6cfef2ceb8d1305727512b16" exitCode=6
Jul 17 22:14:07.528575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.528165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" event={"ID":"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec","Type":"ContainerDied","Data":"d6efab2e2eb29c3a998b3d8d6d85535c46955d4d6cfef2ceb8d1305727512b16"}
Jul 17 22:14:07.528575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.528230    2563 scope.go:117] "RemoveContainer" containerID="aef23cfc43fef62690a863e667b4105650e5ab91021f31d39de2bc62bf633d58"
Jul 17 22:14:07.528575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.528511    2563 scope.go:117] "RemoveContainer" containerID="d6efab2e2eb29c3a998b3d8d6d85535c46955d4d6cfef2ceb8d1305727512b16"
Jul 17 22:14:07.528779 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:07.528755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs_odh-ai-gateway-infra(1f5e2545-a4ad-4fe7-9b80-8605a7c813ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec"
Jul 17 22:14:07.529911 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.529894    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerID="b5bd4de4e1fc0cfdeb7d3a9fcd43d56b20dc351bcffdf9e2a4c06fe1b9aa4613" exitCode=6
Jul 17 22:14:07.530010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.529927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" event={"ID":"c7e438c9-70be-435b-8cc6-30bbbee13d6d","Type":"ContainerDied","Data":"b5bd4de4e1fc0cfdeb7d3a9fcd43d56b20dc351bcffdf9e2a4c06fe1b9aa4613"}
Jul 17 22:14:07.530175 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:07.530162    2563 scope.go:117] "RemoveContainer" containerID="b5bd4de4e1fc0cfdeb7d3a9fcd43d56b20dc351bcffdf9e2a4c06fe1b9aa4613"
Jul 17 22:14:08.539031 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.539000    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerID="11e6975820311817b54d7d993b8ab7c4bf036a2b1585fc8ffab1fca801946191" exitCode=6
Jul 17 22:14:08.539470 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.539083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" event={"ID":"c7e438c9-70be-435b-8cc6-30bbbee13d6d","Type":"ContainerDied","Data":"11e6975820311817b54d7d993b8ab7c4bf036a2b1585fc8ffab1fca801946191"}
Jul 17 22:14:08.539470 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.539137    2563 scope.go:117] "RemoveContainer" containerID="b5bd4de4e1fc0cfdeb7d3a9fcd43d56b20dc351bcffdf9e2a4c06fe1b9aa4613"
Jul 17 22:14:08.539470 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.539250    2563 scope.go:117] "RemoveContainer" containerID="11e6975820311817b54d7d993b8ab7c4bf036a2b1585fc8ffab1fca801946191"
Jul 17 22:14:08.539619 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:08.539512    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb_odh-ai-gateway-infra(c7e438c9-70be-435b-8cc6-30bbbee13d6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d"
Jul 17 22:14:08.553082 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.553062    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"]
Jul 17 22:14:08.695472 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.695444    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:14:08.820654 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.820507    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec-maas-api-service-ca\") pod \"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec\" (UID: \"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec\") "
Jul 17 22:14:08.820937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.820912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" (UID: "1f5e2545-a4ad-4fe7-9b80-8605a7c813ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:08.921727 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:08.921689    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:14:09.545256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.545225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"
Jul 17 22:14:09.545713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.545220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs" event={"ID":"1f5e2545-a4ad-4fe7-9b80-8605a7c813ec","Type":"ContainerDied","Data":"6fd90676435d2a5c451657cd56e9a6d58d1472cb42c4f7129b9ea6200aefcb99"}
Jul 17 22:14:09.545713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.545404    2563 scope.go:117] "RemoveContainer" containerID="d6efab2e2eb29c3a998b3d8d6d85535c46955d4d6cfef2ceb8d1305727512b16"
Jul 17 22:14:09.547104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.547086    2563 scope.go:117] "RemoveContainer" containerID="11e6975820311817b54d7d993b8ab7c4bf036a2b1585fc8ffab1fca801946191"
Jul 17 22:14:09.547312 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:09.547291    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb_odh-ai-gateway-infra(c7e438c9-70be-435b-8cc6-30bbbee13d6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d"
Jul 17 22:14:09.576653 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.576628    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"]
Jul 17 22:14:09.580225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.580202    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-t4hrs"]
Jul 17 22:14:09.902711 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.902633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"]
Jul 17 22:14:09.903037 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.903024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:09.903080 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.903039    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:09.903080 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.903052    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:09.903080 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.903058    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:09.903174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.903131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:09.903174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.903144    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:09.907749 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.907733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:09.910226 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.910200    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-be8c97\"/\"openshift-service-ca.crt\""
Jul 17 22:14:09.911049 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.911026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-be8c97\"/\"test-model-be8c97-kserve-self-signed-certs\""
Jul 17 22:14:09.911134 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.911090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-be8c97\"/\"default-dockercfg-jm2zn\""
Jul 17 22:14:09.911134 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.911108    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-be8c97\"/\"kube-root-ca.crt\""
Jul 17 22:14:09.916273 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:09.916248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"]
Jul 17 22:14:10.030669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.030636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-model-cache\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.030669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.030673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6n7g\" (UniqueName: \"kubernetes.io/projected/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-kube-api-access-c6n7g\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.030902 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.030696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-home\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.030902 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.030777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tls-certs\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.030902 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.030837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-dshm\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.030902 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.030884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tmp-dir\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tls-certs\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132477 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-dshm\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132477 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tmp-dir\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132555 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-model-cache\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132635 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6n7g\" (UniqueName: \"kubernetes.io/projected/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-kube-api-access-c6n7g\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132635 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-home\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132847 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tmp-dir\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132962 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-model-cache\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.132962 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.132952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-home\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.134879 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.134859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-dshm\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.135174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.135153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tls-certs\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.140252 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.140225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6n7g\" (UniqueName: \"kubernetes.io/projected/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-kube-api-access-c6n7g\") pod \"test-model-be8c97-kserve-db94549-kzz76\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") " pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.217632 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.217578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:10.353256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.353232    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"]
Jul 17 22:14:10.355093 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:10.355061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd917144e_09d5_40ba_a9ae_bbe1ae9202ae.slice/crio-1ce3c2030105e84eb110ea98039ccf0eab5bafe04644f7da03efbc21db144d63 WatchSource:0}: Error finding container 1ce3c2030105e84eb110ea98039ccf0eab5bafe04644f7da03efbc21db144d63: Status 404 returned error can't find the container with id 1ce3c2030105e84eb110ea98039ccf0eab5bafe04644f7da03efbc21db144d63
Jul 17 22:14:10.554642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.554539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76" event={"ID":"d917144e-09d5-40ba-a9ae-bbe1ae9202ae","Type":"ContainerStarted","Data":"da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7"}
Jul 17 22:14:10.554642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.554578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76" event={"ID":"d917144e-09d5-40ba-a9ae-bbe1ae9202ae","Type":"ContainerStarted","Data":"1ce3c2030105e84eb110ea98039ccf0eab5bafe04644f7da03efbc21db144d63"}
Jul 17 22:14:10.572832 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.572763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76" podStartSLOduration=1.5727487039999999 podStartE2EDuration="1.572748704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:10.571980878 +0000 UTC m=+2814.484221164" watchObservedRunningTime="2026-07-17 22:14:10.572748704 +0000 UTC m=+2814.484988993"
Jul 17 22:14:10.603248 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:10.603216    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" path="/var/lib/kubelet/pods/1f5e2545-a4ad-4fe7-9b80-8605a7c813ec/volumes"
Jul 17 22:14:20.145774 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.145736    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"]
Jul 17 22:14:20.146239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.146126    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:20.146239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.146137    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:20.146239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.146199    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5e2545-a4ad-4fe7-9b80-8605a7c813ec" containerName="revoke-keys"
Jul 17 22:14:20.150645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.150628    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:20.155986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.155952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"]
Jul 17 22:14:20.217958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.217921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:20.218131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.217973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:20.230713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.230682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:20.329718 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.329686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl\" (UID: \"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:20.430408 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.430380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl\" (UID: \"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:20.431061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.431040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl\" (UID: \"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:20.463574 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.463548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:20.600693 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:20.600659    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeaaf4a_da00_4dec_ab49_f5da30f2d6a2.slice/crio-77fd848ccf9c41f5f6eac7b4a73a56d8e09477585883cf1caa90aad383e3aded WatchSource:0}: Error finding container 77fd848ccf9c41f5f6eac7b4a73a56d8e09477585883cf1caa90aad383e3aded: Status 404 returned error can't find the container with id 77fd848ccf9c41f5f6eac7b4a73a56d8e09477585883cf1caa90aad383e3aded
Jul 17 22:14:20.603443 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.603402    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"]
Jul 17 22:14:20.622351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:20.622326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:14:21.611677 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:21.611643    2563 generic.go:356] "Generic (PLEG): container finished" podID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerID="c87e2ab66add1a6bbfdb3582a5463514cdf1158b08160c8afa1c396ef6c2edb9" exitCode=6
Jul 17 22:14:21.612084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:21.611711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" event={"ID":"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2","Type":"ContainerDied","Data":"c87e2ab66add1a6bbfdb3582a5463514cdf1158b08160c8afa1c396ef6c2edb9"}
Jul 17 22:14:21.612084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:21.611757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" event={"ID":"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2","Type":"ContainerStarted","Data":"77fd848ccf9c41f5f6eac7b4a73a56d8e09477585883cf1caa90aad383e3aded"}
Jul 17 22:14:21.612084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:21.611966    2563 scope.go:117] "RemoveContainer" containerID="c87e2ab66add1a6bbfdb3582a5463514cdf1158b08160c8afa1c396ef6c2edb9"
Jul 17 22:14:22.618616 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:22.618569    2563 generic.go:356] "Generic (PLEG): container finished" podID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerID="f7883d77e13c2d26b212c578409e3916e2f263ccf1c18395964922f13bdc3b7f" exitCode=6
Jul 17 22:14:22.619019 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:22.618655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" event={"ID":"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2","Type":"ContainerDied","Data":"f7883d77e13c2d26b212c578409e3916e2f263ccf1c18395964922f13bdc3b7f"}
Jul 17 22:14:22.619019 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:22.618710    2563 scope.go:117] "RemoveContainer" containerID="c87e2ab66add1a6bbfdb3582a5463514cdf1158b08160c8afa1c396ef6c2edb9"
Jul 17 22:14:22.619019 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:22.618892    2563 scope.go:117] "RemoveContainer" containerID="f7883d77e13c2d26b212c578409e3916e2f263ccf1c18395964922f13bdc3b7f"
Jul 17 22:14:22.619163 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:22.619132    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl_odh-ai-gateway-infra(edeaaf4a-da00-4dec-ab49-f5da30f2d6a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2"
Jul 17 22:14:23.324482 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.324446    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"]
Jul 17 22:14:23.329120 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.329103    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:23.336396 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.336366    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"]
Jul 17 22:14:23.457644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.457614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h\" (UID: \"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:23.558958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.558922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h\" (UID: \"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:23.559565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.559546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h\" (UID: \"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:23.597214 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.597147    2563 scope.go:117] "RemoveContainer" containerID="11e6975820311817b54d7d993b8ab7c4bf036a2b1585fc8ffab1fca801946191"
Jul 17 22:14:23.624960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.624935    2563 scope.go:117] "RemoveContainer" containerID="f7883d77e13c2d26b212c578409e3916e2f263ccf1c18395964922f13bdc3b7f"
Jul 17 22:14:23.625289 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:23.625131    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl_odh-ai-gateway-infra(edeaaf4a-da00-4dec-ab49-f5da30f2d6a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2"
Jul 17 22:14:23.642707 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.642680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:23.785035 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:23.785006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"]
Jul 17 22:14:23.787492 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:23.787457    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd303b0e3_0f4d_405d_a4df_72c9cf8eabc9.slice/crio-fc36653b576af3eed109dd1b3302aaaffd584402b8a7d6257b78b27d2818dfd8 WatchSource:0}: Error finding container fc36653b576af3eed109dd1b3302aaaffd584402b8a7d6257b78b27d2818dfd8: Status 404 returned error can't find the container with id fc36653b576af3eed109dd1b3302aaaffd584402b8a7d6257b78b27d2818dfd8
Jul 17 22:14:24.631286 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.631252    2563 generic.go:356] "Generic (PLEG): container finished" podID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerID="23ef7e3ce75dfb45c727102deab0df3fc41ffa0e04c6f36740fa02391d533bc7" exitCode=6
Jul 17 22:14:24.631768 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.631325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" event={"ID":"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9","Type":"ContainerDied","Data":"23ef7e3ce75dfb45c727102deab0df3fc41ffa0e04c6f36740fa02391d533bc7"}
Jul 17 22:14:24.631768 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.631378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" event={"ID":"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9","Type":"ContainerStarted","Data":"fc36653b576af3eed109dd1b3302aaaffd584402b8a7d6257b78b27d2818dfd8"}
Jul 17 22:14:24.631768 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.631636    2563 scope.go:117] "RemoveContainer" containerID="23ef7e3ce75dfb45c727102deab0df3fc41ffa0e04c6f36740fa02391d533bc7"
Jul 17 22:14:24.633375 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.633353    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerID="7b38a7e4b9e77ab6ae3112c9dad4b366855d759e72530ca37d0dd47d0349b34e" exitCode=6
Jul 17 22:14:24.633479 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.633394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" event={"ID":"c7e438c9-70be-435b-8cc6-30bbbee13d6d","Type":"ContainerDied","Data":"7b38a7e4b9e77ab6ae3112c9dad4b366855d759e72530ca37d0dd47d0349b34e"}
Jul 17 22:14:24.633479 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.633439    2563 scope.go:117] "RemoveContainer" containerID="11e6975820311817b54d7d993b8ab7c4bf036a2b1585fc8ffab1fca801946191"
Jul 17 22:14:24.633704 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:24.633690    2563 scope.go:117] "RemoveContainer" containerID="7b38a7e4b9e77ab6ae3112c9dad4b366855d759e72530ca37d0dd47d0349b34e"
Jul 17 22:14:24.633898 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:24.633877    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb_odh-ai-gateway-infra(c7e438c9-70be-435b-8cc6-30bbbee13d6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d"
Jul 17 22:14:25.642855 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.642770    2563 generic.go:356] "Generic (PLEG): container finished" podID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerID="f90d331e651cf4d07388f502301008bd5f35038f009de0715943d208e059fac7" exitCode=6
Jul 17 22:14:25.642855 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.642808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" event={"ID":"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9","Type":"ContainerDied","Data":"f90d331e651cf4d07388f502301008bd5f35038f009de0715943d208e059fac7"}
Jul 17 22:14:25.642855 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.642855    2563 scope.go:117] "RemoveContainer" containerID="23ef7e3ce75dfb45c727102deab0df3fc41ffa0e04c6f36740fa02391d533bc7"
Jul 17 22:14:25.643374 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.643030    2563 scope.go:117] "RemoveContainer" containerID="f90d331e651cf4d07388f502301008bd5f35038f009de0715943d208e059fac7"
Jul 17 22:14:25.643374 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:25.643249    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h_odh-ai-gateway-infra(d303b0e3-0f4d-405d-a4df-72c9cf8eabc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9"
Jul 17 22:14:25.671738 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.671709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"]
Jul 17 22:14:25.818278 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.818254    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:25.882477 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.882446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e438c9-70be-435b-8cc6-30bbbee13d6d-maas-api-service-ca\") pod \"c7e438c9-70be-435b-8cc6-30bbbee13d6d\" (UID: \"c7e438c9-70be-435b-8cc6-30bbbee13d6d\") "
Jul 17 22:14:25.882841 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.882821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e438c9-70be-435b-8cc6-30bbbee13d6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7e438c9-70be-435b-8cc6-30bbbee13d6d" (UID: "c7e438c9-70be-435b-8cc6-30bbbee13d6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:25.983067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:25.983035    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e438c9-70be-435b-8cc6-30bbbee13d6d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.649011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:26.648978    2563 scope.go:117] "RemoveContainer" containerID="f90d331e651cf4d07388f502301008bd5f35038f009de0715943d208e059fac7"
Jul 17 22:14:26.649427 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:26.649244    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h_odh-ai-gateway-infra(d303b0e3-0f4d-405d-a4df-72c9cf8eabc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9"
Jul 17 22:14:26.650038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:26.650023    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"
Jul 17 22:14:26.650135 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:26.650041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb" event={"ID":"c7e438c9-70be-435b-8cc6-30bbbee13d6d","Type":"ContainerDied","Data":"ac27d092155d3d4b12cc9b519b8937a6abf298487707382c481ea8de758f6a6f"}
Jul 17 22:14:26.650135 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:26.650088    2563 scope.go:117] "RemoveContainer" containerID="7b38a7e4b9e77ab6ae3112c9dad4b366855d759e72530ca37d0dd47d0349b34e"
Jul 17 22:14:26.674522 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:26.674497    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"]
Jul 17 22:14:26.680536 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:26.680515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m5ctb"]
Jul 17 22:14:27.491198 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491157    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"]
Jul 17 22:14:27.491806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491782    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.491914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491809    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.491914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491823    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.491914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491832    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.491914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491855    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.491914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491864    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.492167 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491956    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.492167 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.491973    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:27.496507 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.496490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.500195 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.500171    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b09be9\"/\"openshift-service-ca.crt\""
Jul 17 22:14:27.500195 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.500192    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b09be9\"/\"default-dockercfg-7vx7x\""
Jul 17 22:14:27.500363 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.500195    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b09be9\"/\"kube-root-ca.crt\""
Jul 17 22:14:27.500363 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.500174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b09be9\"/\"test-model-b09be9-kserve-self-signed-certs\""
Jul 17 22:14:27.503980 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.503846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"]
Jul 17 22:14:27.595637 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.595576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-dshm\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.595637 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.595640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-home\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.595852 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.595672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-tmp-dir\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.595852 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.595759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-model-cache\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.595852 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.595785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgj6h\" (UniqueName: \"kubernetes.io/projected/d172c94b-d348-42b2-aea1-764bcfe6d985-kube-api-access-zgj6h\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.595852 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.595810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172c94b-d348-42b2-aea1-764bcfe6d985-tls-certs\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696243 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172c94b-d348-42b2-aea1-764bcfe6d985-tls-certs\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696243 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-dshm\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-home\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-tmp-dir\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-model-cache\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgj6h\" (UniqueName: \"kubernetes.io/projected/d172c94b-d348-42b2-aea1-764bcfe6d985-kube-api-access-zgj6h\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-home\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.696818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-model-cache\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.697047 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.696821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-tmp-dir\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.698793 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.698770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-dshm\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.699061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.699044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172c94b-d348-42b2-aea1-764bcfe6d985-tls-certs\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.704125 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.704100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgj6h\" (UniqueName: \"kubernetes.io/projected/d172c94b-d348-42b2-aea1-764bcfe6d985-kube-api-access-zgj6h\") pod \"test-model-b09be9-kserve-5466686d79-v7wc8\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") " pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.806949 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.806850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:27.941550 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:27.941523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"]
Jul 17 22:14:27.943281 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:27.943252    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd172c94b_d348_42b2_aea1_764bcfe6d985.slice/crio-66e45ec219d4e14456440c7e7cb56289818d2e83febb4dceb5c26bae03215820 WatchSource:0}: Error finding container 66e45ec219d4e14456440c7e7cb56289818d2e83febb4dceb5c26bae03215820: Status 404 returned error can't find the container with id 66e45ec219d4e14456440c7e7cb56289818d2e83febb4dceb5c26bae03215820
Jul 17 22:14:28.609823 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:28.609788    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" path="/var/lib/kubelet/pods/c7e438c9-70be-435b-8cc6-30bbbee13d6d/volumes"
Jul 17 22:14:28.662242 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:28.662203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8" event={"ID":"d172c94b-d348-42b2-aea1-764bcfe6d985","Type":"ContainerStarted","Data":"8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c"}
Jul 17 22:14:28.662385 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:28.662256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8" event={"ID":"d172c94b-d348-42b2-aea1-764bcfe6d985","Type":"ContainerStarted","Data":"66e45ec219d4e14456440c7e7cb56289818d2e83febb4dceb5c26bae03215820"}
Jul 17 22:14:28.678918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:28.678871    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8" podStartSLOduration=1.678854949 podStartE2EDuration="1.678854949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:28.677912157 +0000 UTC m=+2832.590152446" watchObservedRunningTime="2026-07-17 22:14:28.678854949 +0000 UTC m=+2832.591095261"
Jul 17 22:14:37.597405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:37.597370    2563 scope.go:117] "RemoveContainer" containerID="f7883d77e13c2d26b212c578409e3916e2f263ccf1c18395964922f13bdc3b7f"
Jul 17 22:14:37.807766 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:37.807736    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:37.807891 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:37.807783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:37.820504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:37.820479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:38.713703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:38.713668    2563 generic.go:356] "Generic (PLEG): container finished" podID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerID="413bfdcb23b0a87ff74a9754d880c1a51b8928ff76a3e9f8c687411eecab705c" exitCode=6
Jul 17 22:14:38.714148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:38.713742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" event={"ID":"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2","Type":"ContainerDied","Data":"413bfdcb23b0a87ff74a9754d880c1a51b8928ff76a3e9f8c687411eecab705c"}
Jul 17 22:14:38.714148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:38.713803    2563 scope.go:117] "RemoveContainer" containerID="f7883d77e13c2d26b212c578409e3916e2f263ccf1c18395964922f13bdc3b7f"
Jul 17 22:14:38.714148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:38.714079    2563 scope.go:117] "RemoveContainer" containerID="413bfdcb23b0a87ff74a9754d880c1a51b8928ff76a3e9f8c687411eecab705c"
Jul 17 22:14:38.714326 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:38.714302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl_odh-ai-gateway-infra(edeaaf4a-da00-4dec-ab49-f5da30f2d6a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2"
Jul 17 22:14:38.726934 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:38.726914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:14:39.741093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:39.741061    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"]
Jul 17 22:14:39.893965 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:39.893941    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:39.999355 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:39.999271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2-maas-api-service-ca\") pod \"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2\" (UID: \"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2\") "
Jul 17 22:14:39.999658 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:39.999635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" (UID: "edeaaf4a-da00-4dec-ab49-f5da30f2d6a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:40.100433 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:40.100392    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:14:40.727708 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:40.727673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl" event={"ID":"edeaaf4a-da00-4dec-ab49-f5da30f2d6a2","Type":"ContainerDied","Data":"77fd848ccf9c41f5f6eac7b4a73a56d8e09477585883cf1caa90aad383e3aded"}
Jul 17 22:14:40.727708 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:40.727693    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"
Jul 17 22:14:40.727932 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:40.727732    2563 scope.go:117] "RemoveContainer" containerID="413bfdcb23b0a87ff74a9754d880c1a51b8928ff76a3e9f8c687411eecab705c"
Jul 17 22:14:40.743621 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:40.743580    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"]
Jul 17 22:14:40.749210 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:40.748790    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-dn4jl"]
Jul 17 22:14:41.597242 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:41.597201    2563 scope.go:117] "RemoveContainer" containerID="f90d331e651cf4d07388f502301008bd5f35038f009de0715943d208e059fac7"
Jul 17 22:14:42.603126 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:42.603093    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" path="/var/lib/kubelet/pods/edeaaf4a-da00-4dec-ab49-f5da30f2d6a2/volumes"
Jul 17 22:14:42.740494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:42.740466    2563 generic.go:356] "Generic (PLEG): container finished" podID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerID="c3ae7d153aef1e0c013c01c04dec7ce2d449173b7179c58738122eceb7d35c32" exitCode=6
Jul 17 22:14:42.740721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:42.740544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" event={"ID":"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9","Type":"ContainerDied","Data":"c3ae7d153aef1e0c013c01c04dec7ce2d449173b7179c58738122eceb7d35c32"}
Jul 17 22:14:42.740721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:42.740637    2563 scope.go:117] "RemoveContainer" containerID="f90d331e651cf4d07388f502301008bd5f35038f009de0715943d208e059fac7"
Jul 17 22:14:42.740954 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:42.740939    2563 scope.go:117] "RemoveContainer" containerID="c3ae7d153aef1e0c013c01c04dec7ce2d449173b7179c58738122eceb7d35c32"
Jul 17 22:14:42.741197 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:42.741179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h_odh-ai-gateway-infra(d303b0e3-0f4d-405d-a4df-72c9cf8eabc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9"
Jul 17 22:14:43.764666 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:43.764630    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"]
Jul 17 22:14:43.916963 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:43.916939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:43.935326 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:43.935289    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9-maas-api-service-ca\") pod \"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9\" (UID: \"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9\") "
Jul 17 22:14:43.935937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:43.935904    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" (UID: "d303b0e3-0f4d-405d-a4df-72c9cf8eabc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:44.036559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:44.036477    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:14:44.752814 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:44.752782    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"
Jul 17 22:14:44.752991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:44.752814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h" event={"ID":"d303b0e3-0f4d-405d-a4df-72c9cf8eabc9","Type":"ContainerDied","Data":"fc36653b576af3eed109dd1b3302aaaffd584402b8a7d6257b78b27d2818dfd8"}
Jul 17 22:14:44.752991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:44.752880    2563 scope.go:117] "RemoveContainer" containerID="c3ae7d153aef1e0c013c01c04dec7ce2d449173b7179c58738122eceb7d35c32"
Jul 17 22:14:44.767978 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:44.767951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"]
Jul 17 22:14:44.775296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:44.775269    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8s65h"]
Jul 17 22:14:46.607714 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:46.607676    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" path="/var/lib/kubelet/pods/d303b0e3-0f4d-405d-a4df-72c9cf8eabc9/volumes"
Jul 17 22:14:47.962352 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962316    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"]
Jul 17 22:14:47.962896 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962874    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962927    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962955    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962964    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962972    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.962987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.962992    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.963001    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.963100    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.963113    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.963122    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.963132    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e438c9-70be-435b-8cc6-30bbbee13d6d" containerName="revoke-keys"
Jul 17 22:14:47.963330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.963142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:47.967535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.967513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:14:47.969854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.969833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:14:47.971646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:47.971621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"]
Jul 17 22:14:48.068892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.068858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429c045d-9253-480e-b90e-6828d7e16dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt\" (UID: \"429c045d-9253-480e-b90e-6828d7e16dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:14:48.170312 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.170279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429c045d-9253-480e-b90e-6828d7e16dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt\" (UID: \"429c045d-9253-480e-b90e-6828d7e16dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:14:48.170978 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.170960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429c045d-9253-480e-b90e-6828d7e16dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt\" (UID: \"429c045d-9253-480e-b90e-6828d7e16dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:14:48.280499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.280420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:14:48.419098 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.419070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"]
Jul 17 22:14:48.421083 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:48.421047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429c045d_9253_480e_b90e_6828d7e16dab.slice/crio-d7b2ff61accfaf3ab975a4143148e400e983f5ce9ec4f2ffdd3af9cdb501c9c4 WatchSource:0}: Error finding container d7b2ff61accfaf3ab975a4143148e400e983f5ce9ec4f2ffdd3af9cdb501c9c4: Status 404 returned error can't find the container with id d7b2ff61accfaf3ab975a4143148e400e983f5ce9ec4f2ffdd3af9cdb501c9c4
Jul 17 22:14:48.774095 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.774053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" event={"ID":"429c045d-9253-480e-b90e-6828d7e16dab","Type":"ContainerStarted","Data":"460362b58065f9fea4da170c478203b3ce9ac33dfabdb6da552edd925569f784"}
Jul 17 22:14:48.774282 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.774118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" event={"ID":"429c045d-9253-480e-b90e-6828d7e16dab","Type":"ContainerStarted","Data":"d7b2ff61accfaf3ab975a4143148e400e983f5ce9ec4f2ffdd3af9cdb501c9c4"}
Jul 17 22:14:48.791663 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:48.791615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" podStartSLOduration=1.791581917 podStartE2EDuration="1.791581917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:48.789694454 +0000 UTC m=+2852.701934744" watchObservedRunningTime="2026-07-17 22:14:48.791581917 +0000 UTC m=+2852.703822205"
Jul 17 22:14:49.780677 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:49.780644    2563 generic.go:356] "Generic (PLEG): container finished" podID="429c045d-9253-480e-b90e-6828d7e16dab" containerID="460362b58065f9fea4da170c478203b3ce9ac33dfabdb6da552edd925569f784" exitCode=6
Jul 17 22:14:49.781083 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:49.780722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" event={"ID":"429c045d-9253-480e-b90e-6828d7e16dab","Type":"ContainerDied","Data":"460362b58065f9fea4da170c478203b3ce9ac33dfabdb6da552edd925569f784"}
Jul 17 22:14:49.781083 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:49.780912    2563 scope.go:117] "RemoveContainer" containerID="460362b58065f9fea4da170c478203b3ce9ac33dfabdb6da552edd925569f784"
Jul 17 22:14:50.787348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:50.787316    2563 generic.go:356] "Generic (PLEG): container finished" podID="429c045d-9253-480e-b90e-6828d7e16dab" containerID="4ce833122838eaab8d52eabd8eb06628c5feb7fad2632d5f9097f147d240eb08" exitCode=6
Jul 17 22:14:50.787780 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:50.787373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" event={"ID":"429c045d-9253-480e-b90e-6828d7e16dab","Type":"ContainerDied","Data":"4ce833122838eaab8d52eabd8eb06628c5feb7fad2632d5f9097f147d240eb08"}
Jul 17 22:14:50.787780 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:50.787425    2563 scope.go:117] "RemoveContainer" containerID="460362b58065f9fea4da170c478203b3ce9ac33dfabdb6da552edd925569f784"
Jul 17 22:14:50.787780 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:50.787654    2563 scope.go:117] "RemoveContainer" containerID="4ce833122838eaab8d52eabd8eb06628c5feb7fad2632d5f9097f147d240eb08"
Jul 17 22:14:50.787946 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:50.787857    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt_odh-ai-gateway-infra(429c045d-9253-480e-b90e-6828d7e16dab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" podUID="429c045d-9253-480e-b90e-6828d7e16dab"
Jul 17 22:14:51.794272 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:51.794239    2563 scope.go:117] "RemoveContainer" containerID="4ce833122838eaab8d52eabd8eb06628c5feb7fad2632d5f9097f147d240eb08"
Jul 17 22:14:51.794841 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:14:51.794507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt_odh-ai-gateway-infra(429c045d-9253-480e-b90e-6828d7e16dab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" podUID="429c045d-9253-480e-b90e-6828d7e16dab"
Jul 17 22:14:53.416375 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:53.416339    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"]
Jul 17 22:14:54.419283 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.419248    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"]
Jul 17 22:14:54.549194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.549158    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"]
Jul 17 22:14:54.549568 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.549555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:54.549643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.549570    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:54.549684 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.549646    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edeaaf4a-da00-4dec-ab49-f5da30f2d6a2" containerName="revoke-keys"
Jul 17 22:14:54.549684 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.549656    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d303b0e3-0f4d-405d-a4df-72c9cf8eabc9" containerName="revoke-keys"
Jul 17 22:14:54.553923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.553906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:54.558888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.558857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"]
Jul 17 22:14:54.629563 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.629522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba1c78ec-9984-448b-a467-aac0c49317e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c\" (UID: \"ba1c78ec-9984-448b-a467-aac0c49317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:54.731100 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.731006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba1c78ec-9984-448b-a467-aac0c49317e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c\" (UID: \"ba1c78ec-9984-448b-a467-aac0c49317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:54.731674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.731653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba1c78ec-9984-448b-a467-aac0c49317e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c\" (UID: \"ba1c78ec-9984-448b-a467-aac0c49317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:54.866059 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:54.866026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:55.014014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:55.013989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"]
Jul 17 22:14:55.016725 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:14:55.016695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1c78ec_9984_448b_a467_aac0c49317e7.slice/crio-44984ca4ee564723f043c52edd8d0193abdc2b7398827ffad681da374faad357 WatchSource:0}: Error finding container 44984ca4ee564723f043c52edd8d0193abdc2b7398827ffad681da374faad357: Status 404 returned error can't find the container with id 44984ca4ee564723f043c52edd8d0193abdc2b7398827ffad681da374faad357
Jul 17 22:14:55.814433 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:55.814401    2563 generic.go:356] "Generic (PLEG): container finished" podID="ba1c78ec-9984-448b-a467-aac0c49317e7" containerID="5c58f3d7176f435cc647378261ba17a824e1dec9104732ee5cf3643a8a297e9a" exitCode=0
Jul 17 22:14:55.814868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:55.814483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c" event={"ID":"ba1c78ec-9984-448b-a467-aac0c49317e7","Type":"ContainerDied","Data":"5c58f3d7176f435cc647378261ba17a824e1dec9104732ee5cf3643a8a297e9a"}
Jul 17 22:14:55.814868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:55.814538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c" event={"ID":"ba1c78ec-9984-448b-a467-aac0c49317e7","Type":"ContainerStarted","Data":"44984ca4ee564723f043c52edd8d0193abdc2b7398827ffad681da374faad357"}
Jul 17 22:14:56.948864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:56.948842    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:57.053993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:57.053958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba1c78ec-9984-448b-a467-aac0c49317e7-maas-api-service-ca\") pod \"ba1c78ec-9984-448b-a467-aac0c49317e7\" (UID: \"ba1c78ec-9984-448b-a467-aac0c49317e7\") "
Jul 17 22:14:57.054334 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:57.054309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1c78ec-9984-448b-a467-aac0c49317e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba1c78ec-9984-448b-a467-aac0c49317e7" (UID: "ba1c78ec-9984-448b-a467-aac0c49317e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:57.154720 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:57.154639    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba1c78ec-9984-448b-a467-aac0c49317e7-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:14:57.826503 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:57.826472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c"
Jul 17 22:14:57.826503 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:57.826477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c" event={"ID":"ba1c78ec-9984-448b-a467-aac0c49317e7","Type":"ContainerDied","Data":"44984ca4ee564723f043c52edd8d0193abdc2b7398827ffad681da374faad357"}
Jul 17 22:14:57.826738 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:14:57.826526    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44984ca4ee564723f043c52edd8d0193abdc2b7398827ffad681da374faad357"
Jul 17 22:15:00.137159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.137120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"]
Jul 17 22:15:00.137777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.137758    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1c78ec-9984-448b-a467-aac0c49317e7" containerName="revoke-keys"
Jul 17 22:15:00.137832 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.137782    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1c78ec-9984-448b-a467-aac0c49317e7" containerName="revoke-keys"
Jul 17 22:15:00.137908 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.137897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1c78ec-9984-448b-a467-aac0c49317e7" containerName="revoke-keys"
Jul 17 22:15:00.141519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.141503    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"]
Jul 17 22:15:00.141669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.141653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"
Jul 17 22:15:00.144755 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.144735    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"]
Jul 17 22:15:00.144915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.144869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"
Jul 17 22:15:00.148489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.148461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"]
Jul 17 22:15:00.148657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.148567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"
Jul 17 22:15:00.156482 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.156462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"
Jul 17 22:15:00.165034 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.165012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"]
Jul 17 22:15:00.165574 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.165555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"
Jul 17 22:15:00.166944 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.166925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"]
Jul 17 22:15:00.204292 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.204268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"
Jul 17 22:15:00.356643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.356581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"]
Jul 17 22:15:00.358880 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:00.358853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7eaf79_1997_40f7_98b0_1cf4c9e77ed0.slice/crio-3d3e62d88495c5191aeaba20af028e53ce2976764ad3ea2350b35712f817fef2 WatchSource:0}: Error finding container 3d3e62d88495c5191aeaba20af028e53ce2976764ad3ea2350b35712f817fef2: Status 404 returned error can't find the container with id 3d3e62d88495c5191aeaba20af028e53ce2976764ad3ea2350b35712f817fef2
Jul 17 22:15:00.380836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.380813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"]
Jul 17 22:15:00.382870 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:00.382848    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f99c7de_261c_48aa_8c37_6070fa1c06da.slice/crio-8cc3d66d54c053e04e7f2eb59307542949993caae4ea987972ef469a16d9b3c6 WatchSource:0}: Error finding container 8cc3d66d54c053e04e7f2eb59307542949993caae4ea987972ef469a16d9b3c6: Status 404 returned error can't find the container with id 8cc3d66d54c053e04e7f2eb59307542949993caae4ea987972ef469a16d9b3c6
Jul 17 22:15:00.535541 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.535511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"]
Jul 17 22:15:00.536943 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:00.536904    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a783210_9c7e_406a_9250_ea67d641029c.slice/crio-82c8e8962ef6b43475c13a06051d7e2bbebe4b413e9431ea601f6095480692ae WatchSource:0}: Error finding container 82c8e8962ef6b43475c13a06051d7e2bbebe4b413e9431ea601f6095480692ae: Status 404 returned error can't find the container with id 82c8e8962ef6b43475c13a06051d7e2bbebe4b413e9431ea601f6095480692ae
Jul 17 22:15:00.842515 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.842478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh" event={"ID":"7f99c7de-261c-48aa-8c37-6070fa1c06da","Type":"ContainerStarted","Data":"49967041d6b5465397916abdb2dcf1502c29be3d378001916bb8fcfdc6cd8f29"}
Jul 17 22:15:00.842753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.842539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh" event={"ID":"7f99c7de-261c-48aa-8c37-6070fa1c06da","Type":"ContainerStarted","Data":"8cc3d66d54c053e04e7f2eb59307542949993caae4ea987972ef469a16d9b3c6"}
Jul 17 22:15:00.843823 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.843802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv" event={"ID":"8a783210-9c7e-406a-9250-ea67d641029c","Type":"ContainerStarted","Data":"82c8e8962ef6b43475c13a06051d7e2bbebe4b413e9431ea601f6095480692ae"}
Jul 17 22:15:00.845240 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.845216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8" event={"ID":"7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0","Type":"ContainerStarted","Data":"1a758e38f6806476f693616783ff9a04980ca54a54a9a8be5c074ee5ff3bc3b4"}
Jul 17 22:15:00.845352 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.845258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8" event={"ID":"7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0","Type":"ContainerStarted","Data":"3d3e62d88495c5191aeaba20af028e53ce2976764ad3ea2350b35712f817fef2"}
Jul 17 22:15:00.857363 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.857319    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh" podStartSLOduration=0.857306435 podStartE2EDuration="857.306435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:00.855995464 +0000 UTC m=+2864.768235750" watchObservedRunningTime="2026-07-17 22:15:00.857306435 +0000 UTC m=+2864.769546748"
Jul 17 22:15:00.869303 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:00.869265    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8" podStartSLOduration=0.869253061 podStartE2EDuration="869.253061ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:00.868825105 +0000 UTC m=+2864.781065394" watchObservedRunningTime="2026-07-17 22:15:00.869253061 +0000 UTC m=+2864.781493341"
Jul 17 22:15:01.851976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:01.851941    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f99c7de-261c-48aa-8c37-6070fa1c06da" containerID="49967041d6b5465397916abdb2dcf1502c29be3d378001916bb8fcfdc6cd8f29" exitCode=0
Jul 17 22:15:01.852405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:01.852031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh" event={"ID":"7f99c7de-261c-48aa-8c37-6070fa1c06da","Type":"ContainerDied","Data":"49967041d6b5465397916abdb2dcf1502c29be3d378001916bb8fcfdc6cd8f29"}
Jul 17 22:15:01.853668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:01.853637    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a783210-9c7e-406a-9250-ea67d641029c" containerID="f11d329d58b60499109cf7b6da5da1926c8251d582fe40e22c919fee76285042" exitCode=0
Jul 17 22:15:01.853800 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:01.853684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv" event={"ID":"8a783210-9c7e-406a-9250-ea67d641029c","Type":"ContainerDied","Data":"f11d329d58b60499109cf7b6da5da1926c8251d582fe40e22c919fee76285042"}
Jul 17 22:15:01.855178 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:01.855160    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0" containerID="1a758e38f6806476f693616783ff9a04980ca54a54a9a8be5c074ee5ff3bc3b4" exitCode=0
Jul 17 22:15:01.855291 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:01.855232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8" event={"ID":"7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0","Type":"ContainerDied","Data":"1a758e38f6806476f693616783ff9a04980ca54a54a9a8be5c074ee5ff3bc3b4"}
Jul 17 22:15:02.597155 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:02.597120    2563 scope.go:117] "RemoveContainer" containerID="4ce833122838eaab8d52eabd8eb06628c5feb7fad2632d5f9097f147d240eb08"
Jul 17 22:15:03.040909 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.040878    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"
Jul 17 22:15:03.045045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.045022    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"
Jul 17 22:15:03.086887 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.086863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"
Jul 17 22:15:03.866927 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.866895    2563 generic.go:356] "Generic (PLEG): container finished" podID="429c045d-9253-480e-b90e-6828d7e16dab" containerID="7f24b15f44e79d1d90b44b95b6e9de7cbec7dd13decab3873a449f176496593a" exitCode=6
Jul 17 22:15:03.867104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.866972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" event={"ID":"429c045d-9253-480e-b90e-6828d7e16dab","Type":"ContainerDied","Data":"7f24b15f44e79d1d90b44b95b6e9de7cbec7dd13decab3873a449f176496593a"}
Jul 17 22:15:03.867104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.867029    2563 scope.go:117] "RemoveContainer" containerID="4ce833122838eaab8d52eabd8eb06628c5feb7fad2632d5f9097f147d240eb08"
Jul 17 22:15:03.867421 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.867360    2563 scope.go:117] "RemoveContainer" containerID="7f24b15f44e79d1d90b44b95b6e9de7cbec7dd13decab3873a449f176496593a"
Jul 17 22:15:03.867722 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:03.867689    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt_odh-ai-gateway-infra(429c045d-9253-480e-b90e-6828d7e16dab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" podUID="429c045d-9253-480e-b90e-6828d7e16dab"
Jul 17 22:15:03.868832 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.868814    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"
Jul 17 22:15:03.868923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.868832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh" event={"ID":"7f99c7de-261c-48aa-8c37-6070fa1c06da","Type":"ContainerDied","Data":"8cc3d66d54c053e04e7f2eb59307542949993caae4ea987972ef469a16d9b3c6"}
Jul 17 22:15:03.868923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.868874    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cc3d66d54c053e04e7f2eb59307542949993caae4ea987972ef469a16d9b3c6"
Jul 17 22:15:03.870314 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.870292    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv"
Jul 17 22:15:03.870314 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.870309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-28cjv" event={"ID":"8a783210-9c7e-406a-9250-ea67d641029c","Type":"ContainerDied","Data":"82c8e8962ef6b43475c13a06051d7e2bbebe4b413e9431ea601f6095480692ae"}
Jul 17 22:15:03.870516 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.870345    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c8e8962ef6b43475c13a06051d7e2bbebe4b413e9431ea601f6095480692ae"
Jul 17 22:15:03.872283 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.872262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8" event={"ID":"7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0","Type":"ContainerDied","Data":"3d3e62d88495c5191aeaba20af028e53ce2976764ad3ea2350b35712f817fef2"}
Jul 17 22:15:03.872386 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.872300    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d3e62d88495c5191aeaba20af028e53ce2976764ad3ea2350b35712f817fef2"
Jul 17 22:15:03.872386 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:03.872299    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"
Jul 17 22:15:04.923291 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:04.923260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"]
Jul 17 22:15:05.068346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.068323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:15:05.090681 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.090522    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"]
Jul 17 22:15:05.092666 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.092645    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be8c97-29738775-5lvg8"]
Jul 17 22:15:05.227079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.226991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429c045d-9253-480e-b90e-6828d7e16dab-maas-api-service-ca\") pod \"429c045d-9253-480e-b90e-6828d7e16dab\" (UID: \"429c045d-9253-480e-b90e-6828d7e16dab\") "
Jul 17 22:15:05.227369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.227344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429c045d-9253-480e-b90e-6828d7e16dab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "429c045d-9253-480e-b90e-6828d7e16dab" (UID: "429c045d-9253-480e-b90e-6828d7e16dab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:05.328358 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.328323    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429c045d-9253-480e-b90e-6828d7e16dab-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:05.868246 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868194    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"]
Jul 17 22:15:05.868649 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868636    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f99c7de-261c-48aa-8c37-6070fa1c06da" containerName="cleanup"
Jul 17 22:15:05.868705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868651    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f99c7de-261c-48aa-8c37-6070fa1c06da" containerName="cleanup"
Jul 17 22:15:05.868705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868662    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:05.868705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868668    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:05.868705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868693    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:05.868705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:05.868705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868706    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a783210-9c7e-406a-9250-ea67d641029c" containerName="cleanup"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868711    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a783210-9c7e-406a-9250-ea67d641029c" containerName="cleanup"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0" containerName="cleanup"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0" containerName="cleanup"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868793    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a783210-9c7e-406a-9250-ea67d641029c" containerName="cleanup"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868804    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f99c7de-261c-48aa-8c37-6070fa1c06da" containerName="cleanup"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868816    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:05.868943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.868822    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0" containerName="cleanup"
Jul 17 22:15:05.871883 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.871867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:05.877633 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.877567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"]
Jul 17 22:15:05.898915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.898878    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"
Jul 17 22:15:05.898915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.898889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt" event={"ID":"429c045d-9253-480e-b90e-6828d7e16dab","Type":"ContainerDied","Data":"d7b2ff61accfaf3ab975a4143148e400e983f5ce9ec4f2ffdd3af9cdb501c9c4"}
Jul 17 22:15:05.899130 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.899044    2563 scope.go:117] "RemoveContainer" containerID="7f24b15f44e79d1d90b44b95b6e9de7cbec7dd13decab3873a449f176496593a"
Jul 17 22:15:05.920987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.920959    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"]
Jul 17 22:15:05.923260 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:05.923235    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-l2pqt"]
Jul 17 22:15:06.035471 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.035431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf41311-f637-45bd-9690-22026dd938c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m\" (UID: \"4cf41311-f637-45bd-9690-22026dd938c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:06.136331 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.136224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf41311-f637-45bd-9690-22026dd938c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m\" (UID: \"4cf41311-f637-45bd-9690-22026dd938c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:06.136954 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.136928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf41311-f637-45bd-9690-22026dd938c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m\" (UID: \"4cf41311-f637-45bd-9690-22026dd938c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:06.184465 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.184427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:06.531150 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.531120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"]
Jul 17 22:15:06.533934 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:06.533893    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf41311_f637_45bd_9690_22026dd938c3.slice/crio-5f566cdd5353397c6fa0890418ce99cf6dfe4b91e58dc08bae240f223110936d WatchSource:0}: Error finding container 5f566cdd5353397c6fa0890418ce99cf6dfe4b91e58dc08bae240f223110936d: Status 404 returned error can't find the container with id 5f566cdd5353397c6fa0890418ce99cf6dfe4b91e58dc08bae240f223110936d
Jul 17 22:15:06.603657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.603629    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="429c045d-9253-480e-b90e-6828d7e16dab" path="/var/lib/kubelet/pods/429c045d-9253-480e-b90e-6828d7e16dab/volumes"
Jul 17 22:15:06.603944 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.603932    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0" path="/var/lib/kubelet/pods/7b7eaf79-1997-40f7-98b0-1cf4c9e77ed0/volumes"
Jul 17 22:15:06.906227 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.906192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerStarted","Data":"237083cdc72745767b03f4290d2a5d484993488b52f0ba5f308861bcd166d1ab"}
Jul 17 22:15:06.906391 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.906242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerStarted","Data":"5f566cdd5353397c6fa0890418ce99cf6dfe4b91e58dc08bae240f223110936d"}
Jul 17 22:15:06.921905 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:06.921859    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" podStartSLOduration=1.921844863 podStartE2EDuration="1.921844863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:06.920870756 +0000 UTC m=+2870.833111061" watchObservedRunningTime="2026-07-17 22:15:06.921844863 +0000 UTC m=+2870.834085151"
Jul 17 22:15:07.913057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:07.912969    2563 generic.go:356] "Generic (PLEG): container finished" podID="4cf41311-f637-45bd-9690-22026dd938c3" containerID="237083cdc72745767b03f4290d2a5d484993488b52f0ba5f308861bcd166d1ab" exitCode=6
Jul 17 22:15:07.913057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:07.913031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerDied","Data":"237083cdc72745767b03f4290d2a5d484993488b52f0ba5f308861bcd166d1ab"}
Jul 17 22:15:07.913495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:07.913262    2563 scope.go:117] "RemoveContainer" containerID="237083cdc72745767b03f4290d2a5d484993488b52f0ba5f308861bcd166d1ab"
Jul 17 22:15:08.437805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.437770    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76" podUID="d917144e-09d5-40ba-a9ae-bbe1ae9202ae" containerName="main" containerID="cri-o://da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7" gracePeriod=45
Jul 17 22:15:08.692698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.692639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:15:08.859921 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.859887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6n7g\" (UniqueName: \"kubernetes.io/projected/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-kube-api-access-c6n7g\") pod \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") "
Jul 17 22:15:08.860103 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.859933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tls-certs\") pod \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") "
Jul 17 22:15:08.860103 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-dshm\") pod \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") "
Jul 17 22:15:08.860103 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-model-cache\") pod \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") "
Jul 17 22:15:08.860103 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-home\") pod \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") "
Jul 17 22:15:08.860342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tmp-dir\") pod \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\" (UID: \"d917144e-09d5-40ba-a9ae-bbe1ae9202ae\") "
Jul 17 22:15:08.860342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "d917144e-09d5-40ba-a9ae-bbe1ae9202ae" (UID: "d917144e-09d5-40ba-a9ae-bbe1ae9202ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:08.860475 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-home" (OuterVolumeSpecName: "home") pod "d917144e-09d5-40ba-a9ae-bbe1ae9202ae" (UID: "d917144e-09d5-40ba-a9ae-bbe1ae9202ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:08.860534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860472    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:08.860534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.860497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d917144e-09d5-40ba-a9ae-bbe1ae9202ae" (UID: "d917144e-09d5-40ba-a9ae-bbe1ae9202ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:08.862388 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.862369    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-dshm" (OuterVolumeSpecName: "dshm") pod "d917144e-09d5-40ba-a9ae-bbe1ae9202ae" (UID: "d917144e-09d5-40ba-a9ae-bbe1ae9202ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:08.862751 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.862735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d917144e-09d5-40ba-a9ae-bbe1ae9202ae" (UID: "d917144e-09d5-40ba-a9ae-bbe1ae9202ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:15:08.862844 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.862751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-kube-api-access-c6n7g" (OuterVolumeSpecName: "kube-api-access-c6n7g") pod "d917144e-09d5-40ba-a9ae-bbe1ae9202ae" (UID: "d917144e-09d5-40ba-a9ae-bbe1ae9202ae"). InnerVolumeSpecName "kube-api-access-c6n7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:15:08.920211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.920177    2563 generic.go:356] "Generic (PLEG): container finished" podID="4cf41311-f637-45bd-9690-22026dd938c3" containerID="36c0ea7706ae309d2d8eefd52ccd1a0c34398be487d14ad6f7b8d3dbc5658d2e" exitCode=6
Jul 17 22:15:08.920664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.920276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerDied","Data":"36c0ea7706ae309d2d8eefd52ccd1a0c34398be487d14ad6f7b8d3dbc5658d2e"}
Jul 17 22:15:08.920664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.920335    2563 scope.go:117] "RemoveContainer" containerID="237083cdc72745767b03f4290d2a5d484993488b52f0ba5f308861bcd166d1ab"
Jul 17 22:15:08.920664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.920477    2563 scope.go:117] "RemoveContainer" containerID="36c0ea7706ae309d2d8eefd52ccd1a0c34398be487d14ad6f7b8d3dbc5658d2e"
Jul 17 22:15:08.920829 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:08.920760    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m_odh-ai-gateway-infra(4cf41311-f637-45bd-9690-22026dd938c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" podUID="4cf41311-f637-45bd-9690-22026dd938c3"
Jul 17 22:15:08.921768 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.921747    2563 generic.go:356] "Generic (PLEG): container finished" podID="d917144e-09d5-40ba-a9ae-bbe1ae9202ae" containerID="da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7" exitCode=0
Jul 17 22:15:08.921873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.921778    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"
Jul 17 22:15:08.921873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.921816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76" event={"ID":"d917144e-09d5-40ba-a9ae-bbe1ae9202ae","Type":"ContainerDied","Data":"da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7"}
Jul 17 22:15:08.921873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.921848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76" event={"ID":"d917144e-09d5-40ba-a9ae-bbe1ae9202ae","Type":"ContainerDied","Data":"1ce3c2030105e84eb110ea98039ccf0eab5bafe04644f7da03efbc21db144d63"}
Jul 17 22:15:08.932257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.932233    2563 scope.go:117] "RemoveContainer" containerID="da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7"
Jul 17 22:15:08.940309 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.940291    2563 scope.go:117] "RemoveContainer" containerID="da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7"
Jul 17 22:15:08.940533 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:08.940517    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7\": container with ID starting with da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7 not found: ID does not exist" containerID="da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7"
Jul 17 22:15:08.940578 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.940543    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7"} err="failed to get container status \"da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7\": rpc error: code = NotFound desc = could not find container \"da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7\": container with ID starting with da3d18adf246224e4c34b7a8637cf9b0aac0636bb496928c6558215c59f493f7 not found: ID does not exist"
Jul 17 22:15:08.947050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.946993    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"]
Jul 17 22:15:08.952583 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.952562    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-be8c97/test-model-be8c97-kserve-db94549-kzz76"]
Jul 17 22:15:08.961779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.961759    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:08.961868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.961784    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6n7g\" (UniqueName: \"kubernetes.io/projected/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-kube-api-access-c6n7g\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:08.961868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.961798    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:08.961868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.961812    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:08.961868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:08.961824    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d917144e-09d5-40ba-a9ae-bbe1ae9202ae-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.439788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.439752    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8" podUID="d172c94b-d348-42b2-aea1-764bcfe6d985" containerName="main" containerID="cri-o://8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c" gracePeriod=45
Jul 17 22:15:09.694145 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.694088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:15:09.768777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.768747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-tmp-dir\") pod \"d172c94b-d348-42b2-aea1-764bcfe6d985\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") "
Jul 17 22:15:09.769011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.768821    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgj6h\" (UniqueName: \"kubernetes.io/projected/d172c94b-d348-42b2-aea1-764bcfe6d985-kube-api-access-zgj6h\") pod \"d172c94b-d348-42b2-aea1-764bcfe6d985\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") "
Jul 17 22:15:09.769011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.768864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-model-cache\") pod \"d172c94b-d348-42b2-aea1-764bcfe6d985\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") "
Jul 17 22:15:09.769011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.768884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-dshm\") pod \"d172c94b-d348-42b2-aea1-764bcfe6d985\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") "
Jul 17 22:15:09.769011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.768902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-home\") pod \"d172c94b-d348-42b2-aea1-764bcfe6d985\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") "
Jul 17 22:15:09.769235 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d172c94b-d348-42b2-aea1-764bcfe6d985" (UID: "d172c94b-d348-42b2-aea1-764bcfe6d985"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:09.769235 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172c94b-d348-42b2-aea1-764bcfe6d985-tls-certs\") pod \"d172c94b-d348-42b2-aea1-764bcfe6d985\" (UID: \"d172c94b-d348-42b2-aea1-764bcfe6d985\") "
Jul 17 22:15:09.769235 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-model-cache" (OuterVolumeSpecName: "model-cache") pod "d172c94b-d348-42b2-aea1-764bcfe6d985" (UID: "d172c94b-d348-42b2-aea1-764bcfe6d985"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:09.769235 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-home" (OuterVolumeSpecName: "home") pod "d172c94b-d348-42b2-aea1-764bcfe6d985" (UID: "d172c94b-d348-42b2-aea1-764bcfe6d985"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:09.769381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769340    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.769381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769358    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.769381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.769370    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.771269 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.771242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172c94b-d348-42b2-aea1-764bcfe6d985-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d172c94b-d348-42b2-aea1-764bcfe6d985" (UID: "d172c94b-d348-42b2-aea1-764bcfe6d985"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:15:09.771269 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.771258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-dshm" (OuterVolumeSpecName: "dshm") pod "d172c94b-d348-42b2-aea1-764bcfe6d985" (UID: "d172c94b-d348-42b2-aea1-764bcfe6d985"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:09.771413 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.771274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d172c94b-d348-42b2-aea1-764bcfe6d985-kube-api-access-zgj6h" (OuterVolumeSpecName: "kube-api-access-zgj6h") pod "d172c94b-d348-42b2-aea1-764bcfe6d985" (UID: "d172c94b-d348-42b2-aea1-764bcfe6d985"). InnerVolumeSpecName "kube-api-access-zgj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:15:09.870416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.870384    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgj6h\" (UniqueName: \"kubernetes.io/projected/d172c94b-d348-42b2-aea1-764bcfe6d985-kube-api-access-zgj6h\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.870416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.870412    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172c94b-d348-42b2-aea1-764bcfe6d985-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.870416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.870423    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172c94b-d348-42b2-aea1-764bcfe6d985-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:09.934521 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.934493    2563 scope.go:117] "RemoveContainer" containerID="36c0ea7706ae309d2d8eefd52ccd1a0c34398be487d14ad6f7b8d3dbc5658d2e"
Jul 17 22:15:09.935010 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:09.934732    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m_odh-ai-gateway-infra(4cf41311-f637-45bd-9690-22026dd938c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" podUID="4cf41311-f637-45bd-9690-22026dd938c3"
Jul 17 22:15:09.936777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.936754    2563 generic.go:356] "Generic (PLEG): container finished" podID="d172c94b-d348-42b2-aea1-764bcfe6d985" containerID="8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c" exitCode=0
Jul 17 22:15:09.936893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.936794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8" event={"ID":"d172c94b-d348-42b2-aea1-764bcfe6d985","Type":"ContainerDied","Data":"8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c"}
Jul 17 22:15:09.936893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.936800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"
Jul 17 22:15:09.936893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.936816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8" event={"ID":"d172c94b-d348-42b2-aea1-764bcfe6d985","Type":"ContainerDied","Data":"66e45ec219d4e14456440c7e7cb56289818d2e83febb4dceb5c26bae03215820"}
Jul 17 22:15:09.936893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.936831    2563 scope.go:117] "RemoveContainer" containerID="8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c"
Jul 17 22:15:09.946446 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.946422    2563 scope.go:117] "RemoveContainer" containerID="8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c"
Jul 17 22:15:09.946702 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:09.946683    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c\": container with ID starting with 8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c not found: ID does not exist" containerID="8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c"
Jul 17 22:15:09.946769 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.946710    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c"} err="failed to get container status \"8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c\": rpc error: code = NotFound desc = could not find container \"8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c\": container with ID starting with 8731cf164a7b2ab78c1fb0896e4fe4be3cf1111d6e274fbc77d7a6d8f8bafb9c not found: ID does not exist"
Jul 17 22:15:09.961924 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.961897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"]
Jul 17 22:15:09.967471 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:09.967451    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b09be9/test-model-b09be9-kserve-5466686d79-v7wc8"]
Jul 17 22:15:10.602342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:10.602309    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d172c94b-d348-42b2-aea1-764bcfe6d985" path="/var/lib/kubelet/pods/d172c94b-d348-42b2-aea1-764bcfe6d985/volumes"
Jul 17 22:15:10.602697 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:10.602684    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d917144e-09d5-40ba-a9ae-bbe1ae9202ae" path="/var/lib/kubelet/pods/d917144e-09d5-40ba-a9ae-bbe1ae9202ae/volumes"
Jul 17 22:15:20.191055 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191018    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"]
Jul 17 22:15:20.191454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d172c94b-d348-42b2-aea1-764bcfe6d985" containerName="main"
Jul 17 22:15:20.191454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191392    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d172c94b-d348-42b2-aea1-764bcfe6d985" containerName="main"
Jul 17 22:15:20.191454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191410    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:20.191454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191416    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:20.191454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191425    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d917144e-09d5-40ba-a9ae-bbe1ae9202ae" containerName="main"
Jul 17 22:15:20.191454 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191431    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d917144e-09d5-40ba-a9ae-bbe1ae9202ae" containerName="main"
Jul 17 22:15:20.191667 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191488    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d172c94b-d348-42b2-aea1-764bcfe6d985" containerName="main"
Jul 17 22:15:20.191667 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="429c045d-9253-480e-b90e-6828d7e16dab" containerName="revoke-keys"
Jul 17 22:15:20.191667 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.191501    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d917144e-09d5-40ba-a9ae-bbe1ae9202ae" containerName="main"
Jul 17 22:15:20.194671 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.194652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:20.202045 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.202010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"]
Jul 17 22:15:20.266405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.266365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c02142-9625-40c0-bbc1-7f7d086b6868-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls\" (UID: \"c5c02142-9625-40c0-bbc1-7f7d086b6868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:20.366877 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.366840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c02142-9625-40c0-bbc1-7f7d086b6868-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls\" (UID: \"c5c02142-9625-40c0-bbc1-7f7d086b6868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:20.367444 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.367423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c02142-9625-40c0-bbc1-7f7d086b6868-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls\" (UID: \"c5c02142-9625-40c0-bbc1-7f7d086b6868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:20.510361 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.510272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:20.674254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.674227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"]
Jul 17 22:15:20.676670 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:20.676641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c02142_9625_40c0_bbc1_7f7d086b6868.slice/crio-c77db67c024c8561e6b2eb353ad317e81c4fd7532e3067f0c22959b793c5036f WatchSource:0}: Error finding container c77db67c024c8561e6b2eb353ad317e81c4fd7532e3067f0c22959b793c5036f: Status 404 returned error can't find the container with id c77db67c024c8561e6b2eb353ad317e81c4fd7532e3067f0c22959b793c5036f
Jul 17 22:15:20.691583 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.691555    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"]
Jul 17 22:15:20.696254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.696226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:20.702105 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.702076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"]
Jul 17 22:15:20.770489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.770407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8487662-bff1-4cfe-9f61-551a03d48a3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b09be9-phwng\" (UID: \"f8487662-bff1-4cfe-9f61-551a03d48a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:20.871043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.870849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8487662-bff1-4cfe-9f61-551a03d48a3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b09be9-phwng\" (UID: \"f8487662-bff1-4cfe-9f61-551a03d48a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:20.871538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.871519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8487662-bff1-4cfe-9f61-551a03d48a3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b09be9-phwng\" (UID: \"f8487662-bff1-4cfe-9f61-551a03d48a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:20.999805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.999768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerStarted","Data":"8dc5ed3f5956a714927b612f8bf87c57a7b0560d806642c7a4f50acadfc4d27c"}
Jul 17 22:15:20.999974 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:20.999824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerStarted","Data":"c77db67c024c8561e6b2eb353ad317e81c4fd7532e3067f0c22959b793c5036f"}
Jul 17 22:15:21.022347 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:21.022249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" podStartSLOduration=1.022235609 podStartE2EDuration="1.022235609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:21.02072248 +0000 UTC m=+2884.932962769" watchObservedRunningTime="2026-07-17 22:15:21.022235609 +0000 UTC m=+2884.934475898"
Jul 17 22:15:21.026772 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:21.026752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:21.171546 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:21.171522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"]
Jul 17 22:15:21.173255 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:21.173226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8487662_bff1_4cfe_9f61_551a03d48a3c.slice/crio-6c65e600780e989db18e5813e3f1b053377428731213051441041289751f2d95 WatchSource:0}: Error finding container 6c65e600780e989db18e5813e3f1b053377428731213051441041289751f2d95: Status 404 returned error can't find the container with id 6c65e600780e989db18e5813e3f1b053377428731213051441041289751f2d95
Jul 17 22:15:22.006301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:22.006263    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerID="8dc5ed3f5956a714927b612f8bf87c57a7b0560d806642c7a4f50acadfc4d27c" exitCode=6
Jul 17 22:15:22.006753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:22.006350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerDied","Data":"8dc5ed3f5956a714927b612f8bf87c57a7b0560d806642c7a4f50acadfc4d27c"}
Jul 17 22:15:22.006753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:22.006547    2563 scope.go:117] "RemoveContainer" containerID="8dc5ed3f5956a714927b612f8bf87c57a7b0560d806642c7a4f50acadfc4d27c"
Jul 17 22:15:22.007955 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:22.007931    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8487662-bff1-4cfe-9f61-551a03d48a3c" containerID="2333889531a145e505269b6cdefd79a0eb97bbbd02778fd3e3ec6513429b6f34" exitCode=0
Jul 17 22:15:22.008054 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:22.008006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng" event={"ID":"f8487662-bff1-4cfe-9f61-551a03d48a3c","Type":"ContainerDied","Data":"2333889531a145e505269b6cdefd79a0eb97bbbd02778fd3e3ec6513429b6f34"}
Jul 17 22:15:22.008054 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:22.008044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng" event={"ID":"f8487662-bff1-4cfe-9f61-551a03d48a3c","Type":"ContainerStarted","Data":"6c65e600780e989db18e5813e3f1b053377428731213051441041289751f2d95"}
Jul 17 22:15:23.015373 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.015335    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerID="edb340c608f2fbf3f12dfaf264f30506c05a3c616831d8ac43fc36d21e8d3543" exitCode=6
Jul 17 22:15:23.015998 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.015422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerDied","Data":"edb340c608f2fbf3f12dfaf264f30506c05a3c616831d8ac43fc36d21e8d3543"}
Jul 17 22:15:23.015998 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.015478    2563 scope.go:117] "RemoveContainer" containerID="8dc5ed3f5956a714927b612f8bf87c57a7b0560d806642c7a4f50acadfc4d27c"
Jul 17 22:15:23.015998 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.015735    2563 scope.go:117] "RemoveContainer" containerID="edb340c608f2fbf3f12dfaf264f30506c05a3c616831d8ac43fc36d21e8d3543"
Jul 17 22:15:23.015998 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:23.015961    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls_odh-ai-gateway-infra(c5c02142-9625-40c0-bbc1-7f7d086b6868)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868"
Jul 17 22:15:23.160517 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.160491    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:23.187088 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.187057    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8487662-bff1-4cfe-9f61-551a03d48a3c-maas-api-service-ca\") pod \"f8487662-bff1-4cfe-9f61-551a03d48a3c\" (UID: \"f8487662-bff1-4cfe-9f61-551a03d48a3c\") "
Jul 17 22:15:23.187372 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.187350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8487662-bff1-4cfe-9f61-551a03d48a3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8487662-bff1-4cfe-9f61-551a03d48a3c" (UID: "f8487662-bff1-4cfe-9f61-551a03d48a3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:23.288394 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.288309    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8487662-bff1-4cfe-9f61-551a03d48a3c-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:23.361220 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.361184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"]
Jul 17 22:15:23.361622 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.361609    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8487662-bff1-4cfe-9f61-551a03d48a3c" containerName="revoke-keys"
Jul 17 22:15:23.361622 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.361623    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8487662-bff1-4cfe-9f61-551a03d48a3c" containerName="revoke-keys"
Jul 17 22:15:23.361723 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.361681    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8487662-bff1-4cfe-9f61-551a03d48a3c" containerName="revoke-keys"
Jul 17 22:15:23.364782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.364765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:23.371046 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.371020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"]
Jul 17 22:15:23.389458 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.389434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f67b58-ed08-46d8-a7f4-16105605eaba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w\" (UID: \"31f67b58-ed08-46d8-a7f4-16105605eaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:23.490044 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.490007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f67b58-ed08-46d8-a7f4-16105605eaba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w\" (UID: \"31f67b58-ed08-46d8-a7f4-16105605eaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:23.490639 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.490583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f67b58-ed08-46d8-a7f4-16105605eaba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w\" (UID: \"31f67b58-ed08-46d8-a7f4-16105605eaba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:23.677222 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.677197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:23.812276 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:23.812245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"]
Jul 17 22:15:23.814218 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:23.814186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f67b58_ed08_46d8_a7f4_16105605eaba.slice/crio-be6bbf4038b92b7e275f582e9331fc58ed86be62adf5492e361e577a47de0f56 WatchSource:0}: Error finding container be6bbf4038b92b7e275f582e9331fc58ed86be62adf5492e361e577a47de0f56: Status 404 returned error can't find the container with id be6bbf4038b92b7e275f582e9331fc58ed86be62adf5492e361e577a47de0f56
Jul 17 22:15:24.021400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:24.021318    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng"
Jul 17 22:15:24.021400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:24.021358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b09be9-phwng" event={"ID":"f8487662-bff1-4cfe-9f61-551a03d48a3c","Type":"ContainerDied","Data":"6c65e600780e989db18e5813e3f1b053377428731213051441041289751f2d95"}
Jul 17 22:15:24.021885 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:24.021403    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c65e600780e989db18e5813e3f1b053377428731213051441041289751f2d95"
Jul 17 22:15:24.022718 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:24.022688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" event={"ID":"31f67b58-ed08-46d8-a7f4-16105605eaba","Type":"ContainerStarted","Data":"be6bbf4038b92b7e275f582e9331fc58ed86be62adf5492e361e577a47de0f56"}
Jul 17 22:15:24.024519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:24.024501    2563 scope.go:117] "RemoveContainer" containerID="edb340c608f2fbf3f12dfaf264f30506c05a3c616831d8ac43fc36d21e8d3543"
Jul 17 22:15:24.024784 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:24.024760    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls_odh-ai-gateway-infra(c5c02142-9625-40c0-bbc1-7f7d086b6868)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868"
Jul 17 22:15:24.589273 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:24.589229    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f67b58_ed08_46d8_a7f4_16105605eaba.slice/crio-conmon-a458d879f416b249a17b3d55718460fb775503bd8ce2e18f729189cae323d0bc.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 22:15:24.589417 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:24.589327    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f67b58_ed08_46d8_a7f4_16105605eaba.slice/crio-a458d879f416b249a17b3d55718460fb775503bd8ce2e18f729189cae323d0bc.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 22:15:24.596864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:24.596841    2563 scope.go:117] "RemoveContainer" containerID="36c0ea7706ae309d2d8eefd52ccd1a0c34398be487d14ad6f7b8d3dbc5658d2e"
Jul 17 22:15:25.033870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:25.033828    2563 generic.go:356] "Generic (PLEG): container finished" podID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerID="a458d879f416b249a17b3d55718460fb775503bd8ce2e18f729189cae323d0bc" exitCode=6
Jul 17 22:15:25.034320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:25.033919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" event={"ID":"31f67b58-ed08-46d8-a7f4-16105605eaba","Type":"ContainerDied","Data":"a458d879f416b249a17b3d55718460fb775503bd8ce2e18f729189cae323d0bc"}
Jul 17 22:15:25.034320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:25.034135    2563 scope.go:117] "RemoveContainer" containerID="a458d879f416b249a17b3d55718460fb775503bd8ce2e18f729189cae323d0bc"
Jul 17 22:15:25.035905 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:25.035881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerStarted","Data":"ea8611377a46dcbd4a561a920dfa54bd1026df7eaaaadafca83e515aecb3d9e1"}
Jul 17 22:15:26.043738 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.043710    2563 generic.go:356] "Generic (PLEG): container finished" podID="4cf41311-f637-45bd-9690-22026dd938c3" containerID="ea8611377a46dcbd4a561a920dfa54bd1026df7eaaaadafca83e515aecb3d9e1" exitCode=6
Jul 17 22:15:26.044195 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.043785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerDied","Data":"ea8611377a46dcbd4a561a920dfa54bd1026df7eaaaadafca83e515aecb3d9e1"}
Jul 17 22:15:26.044195 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.043862    2563 scope.go:117] "RemoveContainer" containerID="36c0ea7706ae309d2d8eefd52ccd1a0c34398be487d14ad6f7b8d3dbc5658d2e"
Jul 17 22:15:26.044195 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.044190    2563 scope.go:117] "RemoveContainer" containerID="ea8611377a46dcbd4a561a920dfa54bd1026df7eaaaadafca83e515aecb3d9e1"
Jul 17 22:15:26.044470 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:26.044448    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m_odh-ai-gateway-infra(4cf41311-f637-45bd-9690-22026dd938c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" podUID="4cf41311-f637-45bd-9690-22026dd938c3"
Jul 17 22:15:26.046079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.046053    2563 generic.go:356] "Generic (PLEG): container finished" podID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerID="c9b5b2341de28c020ea8b500a91429a68ad6d8d6bb7de9f033103dd4548b0f9f" exitCode=6
Jul 17 22:15:26.046209 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.046164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" event={"ID":"31f67b58-ed08-46d8-a7f4-16105605eaba","Type":"ContainerDied","Data":"c9b5b2341de28c020ea8b500a91429a68ad6d8d6bb7de9f033103dd4548b0f9f"}
Jul 17 22:15:26.046519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.046499    2563 scope.go:117] "RemoveContainer" containerID="c9b5b2341de28c020ea8b500a91429a68ad6d8d6bb7de9f033103dd4548b0f9f"
Jul 17 22:15:26.046762 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:26.046737    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w_odh-ai-gateway-infra(31f67b58-ed08-46d8-a7f4-16105605eaba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba"
Jul 17 22:15:26.056446 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.056431    2563 scope.go:117] "RemoveContainer" containerID="a458d879f416b249a17b3d55718460fb775503bd8ce2e18f729189cae323d0bc"
Jul 17 22:15:26.070763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:26.070736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"]
Jul 17 22:15:27.055147 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:27.055119    2563 scope.go:117] "RemoveContainer" containerID="c9b5b2341de28c020ea8b500a91429a68ad6d8d6bb7de9f033103dd4548b0f9f"
Jul 17 22:15:27.055590 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:27.055287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w_odh-ai-gateway-infra(31f67b58-ed08-46d8-a7f4-16105605eaba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba"
Jul 17 22:15:27.199161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:27.199132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:27.220203 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:27.220174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf41311-f637-45bd-9690-22026dd938c3-maas-api-service-ca\") pod \"4cf41311-f637-45bd-9690-22026dd938c3\" (UID: \"4cf41311-f637-45bd-9690-22026dd938c3\") "
Jul 17 22:15:27.220555 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:27.220535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf41311-f637-45bd-9690-22026dd938c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cf41311-f637-45bd-9690-22026dd938c3" (UID: "4cf41311-f637-45bd-9690-22026dd938c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:27.320965 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:27.320882    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf41311-f637-45bd-9690-22026dd938c3-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:28.061163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:28.061128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m" event={"ID":"4cf41311-f637-45bd-9690-22026dd938c3","Type":"ContainerDied","Data":"5f566cdd5353397c6fa0890418ce99cf6dfe4b91e58dc08bae240f223110936d"}
Jul 17 22:15:28.061617 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:28.061172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"
Jul 17 22:15:28.061617 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:28.061194    2563 scope.go:117] "RemoveContainer" containerID="ea8611377a46dcbd4a561a920dfa54bd1026df7eaaaadafca83e515aecb3d9e1"
Jul 17 22:15:28.081621 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:28.081575    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"]
Jul 17 22:15:28.085113 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:28.085090    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9k78m"]
Jul 17 22:15:28.603131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:28.603099    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf41311-f637-45bd-9690-22026dd938c3" path="/var/lib/kubelet/pods/4cf41311-f637-45bd-9690-22026dd938c3/volumes"
Jul 17 22:15:31.087497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:31.087464    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"]
Jul 17 22:15:31.093032 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:31.093000    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b09be9-29738775-sflxh"]
Jul 17 22:15:32.602539 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:32.602508    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f99c7de-261c-48aa-8c37-6070fa1c06da" path="/var/lib/kubelet/pods/7f99c7de-261c-48aa-8c37-6070fa1c06da/volumes"
Jul 17 22:15:35.596904 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:35.596873    2563 scope.go:117] "RemoveContainer" containerID="edb340c608f2fbf3f12dfaf264f30506c05a3c616831d8ac43fc36d21e8d3543"
Jul 17 22:15:36.104416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:36.104379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerStarted","Data":"1584220f026e86fe19cbbce839c7c5639f2c0b26a1c1b8b24771bfc9b41a7b2d"}
Jul 17 22:15:37.111407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:37.111374    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerID="1584220f026e86fe19cbbce839c7c5639f2c0b26a1c1b8b24771bfc9b41a7b2d" exitCode=6
Jul 17 22:15:37.111822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:37.111443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerDied","Data":"1584220f026e86fe19cbbce839c7c5639f2c0b26a1c1b8b24771bfc9b41a7b2d"}
Jul 17 22:15:37.111822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:37.111514    2563 scope.go:117] "RemoveContainer" containerID="edb340c608f2fbf3f12dfaf264f30506c05a3c616831d8ac43fc36d21e8d3543"
Jul 17 22:15:37.111822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:37.111802    2563 scope.go:117] "RemoveContainer" containerID="1584220f026e86fe19cbbce839c7c5639f2c0b26a1c1b8b24771bfc9b41a7b2d"
Jul 17 22:15:37.112044 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:37.112026    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls_odh-ai-gateway-infra(c5c02142-9625-40c0-bbc1-7f7d086b6868)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868"
Jul 17 22:15:37.135712 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:37.135687    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"]
Jul 17 22:15:38.010704 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.010669    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"]
Jul 17 22:15:38.011067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011055    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011069    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011080    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011087    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011100    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011105    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011191    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.011270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.011198    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:15:38.015214 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.015198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:38.020922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.020892    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"]
Jul 17 22:15:38.109428 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.109389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf\" (UID: \"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:38.209918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.209878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf\" (UID: \"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:38.210756 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.210727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf\" (UID: \"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:38.264587 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.264533    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:38.310451 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.310418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c02142-9625-40c0-bbc1-7f7d086b6868-maas-api-service-ca\") pod \"c5c02142-9625-40c0-bbc1-7f7d086b6868\" (UID: \"c5c02142-9625-40c0-bbc1-7f7d086b6868\") "
Jul 17 22:15:38.310810 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.310790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c02142-9625-40c0-bbc1-7f7d086b6868-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5c02142-9625-40c0-bbc1-7f7d086b6868" (UID: "c5c02142-9625-40c0-bbc1-7f7d086b6868"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:38.326963 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.326937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:38.411896 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.411861    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c02142-9625-40c0-bbc1-7f7d086b6868-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:38.463621 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.463564    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"]
Jul 17 22:15:38.466041 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:15:38.466013    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea80ee5b_d505_4db3_a85a_a8e73bbc54e9.slice/crio-7d71bfcf474e286fae2e111e354bd69a6a2c18231bad1d9da6e4a97da8290d0a WatchSource:0}: Error finding container 7d71bfcf474e286fae2e111e354bd69a6a2c18231bad1d9da6e4a97da8290d0a: Status 404 returned error can't find the container with id 7d71bfcf474e286fae2e111e354bd69a6a2c18231bad1d9da6e4a97da8290d0a
Jul 17 22:15:38.597079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:38.596999    2563 scope.go:117] "RemoveContainer" containerID="c9b5b2341de28c020ea8b500a91429a68ad6d8d6bb7de9f033103dd4548b0f9f"
Jul 17 22:15:39.124004 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.123974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"
Jul 17 22:15:39.124166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.124010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls" event={"ID":"c5c02142-9625-40c0-bbc1-7f7d086b6868","Type":"ContainerDied","Data":"c77db67c024c8561e6b2eb353ad317e81c4fd7532e3067f0c22959b793c5036f"}
Jul 17 22:15:39.124166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.124061    2563 scope.go:117] "RemoveContainer" containerID="1584220f026e86fe19cbbce839c7c5639f2c0b26a1c1b8b24771bfc9b41a7b2d"
Jul 17 22:15:39.126801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.126733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" event={"ID":"31f67b58-ed08-46d8-a7f4-16105605eaba","Type":"ContainerStarted","Data":"74266718ae75184371925823f4c3b3a7ca26ee9130ed9c21b3568c9fa6243857"}
Jul 17 22:15:39.128311 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.128274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" event={"ID":"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9","Type":"ContainerStarted","Data":"6cb2a6187661bc25ce23ad9a43bf7608c20493526264d0f2a35c90b0ad4ccfaf"}
Jul 17 22:15:39.128422 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.128328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" event={"ID":"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9","Type":"ContainerStarted","Data":"7d71bfcf474e286fae2e111e354bd69a6a2c18231bad1d9da6e4a97da8290d0a"}
Jul 17 22:15:39.139959 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.139936    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"]
Jul 17 22:15:39.143575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.143555    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-w2lls"]
Jul 17 22:15:39.158560 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.158516    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" podStartSLOduration=16.158500542 podStartE2EDuration="16.158500542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:39.156405038 +0000 UTC m=+2903.068645326" watchObservedRunningTime="2026-07-17 22:15:39.158500542 +0000 UTC m=+2903.070740831"
Jul 17 22:15:39.171009 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:39.170967    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" podStartSLOduration=2.170953244 podStartE2EDuration="2.170953244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:39.169345392 +0000 UTC m=+2903.081585683" watchObservedRunningTime="2026-07-17 22:15:39.170953244 +0000 UTC m=+2903.083193533"
Jul 17 22:15:40.140093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.140060    2563 generic.go:356] "Generic (PLEG): container finished" podID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerID="74266718ae75184371925823f4c3b3a7ca26ee9130ed9c21b3568c9fa6243857" exitCode=6
Jul 17 22:15:40.140572 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.140144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" event={"ID":"31f67b58-ed08-46d8-a7f4-16105605eaba","Type":"ContainerDied","Data":"74266718ae75184371925823f4c3b3a7ca26ee9130ed9c21b3568c9fa6243857"}
Jul 17 22:15:40.140572 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.140247    2563 scope.go:117] "RemoveContainer" containerID="c9b5b2341de28c020ea8b500a91429a68ad6d8d6bb7de9f033103dd4548b0f9f"
Jul 17 22:15:40.140572 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.140525    2563 scope.go:117] "RemoveContainer" containerID="74266718ae75184371925823f4c3b3a7ca26ee9130ed9c21b3568c9fa6243857"
Jul 17 22:15:40.140765 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:40.140746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w_odh-ai-gateway-infra(31f67b58-ed08-46d8-a7f4-16105605eaba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba"
Jul 17 22:15:40.141912 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.141875    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerID="6cb2a6187661bc25ce23ad9a43bf7608c20493526264d0f2a35c90b0ad4ccfaf" exitCode=6
Jul 17 22:15:40.142031 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.141957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" event={"ID":"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9","Type":"ContainerDied","Data":"6cb2a6187661bc25ce23ad9a43bf7608c20493526264d0f2a35c90b0ad4ccfaf"}
Jul 17 22:15:40.142105 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.142079    2563 scope.go:117] "RemoveContainer" containerID="6cb2a6187661bc25ce23ad9a43bf7608c20493526264d0f2a35c90b0ad4ccfaf"
Jul 17 22:15:40.166294 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.166255    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"]
Jul 17 22:15:40.602687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:40.602643    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" path="/var/lib/kubelet/pods/c5c02142-9625-40c0-bbc1-7f7d086b6868/volumes"
Jul 17 22:15:41.151481 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.151453    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerID="7e1bc59860e62d03e660e5069e31408a77d331955b41bcda492f3f2c1aea0934" exitCode=6
Jul 17 22:15:41.151897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.151505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" event={"ID":"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9","Type":"ContainerDied","Data":"7e1bc59860e62d03e660e5069e31408a77d331955b41bcda492f3f2c1aea0934"}
Jul 17 22:15:41.151897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.151562    2563 scope.go:117] "RemoveContainer" containerID="6cb2a6187661bc25ce23ad9a43bf7608c20493526264d0f2a35c90b0ad4ccfaf"
Jul 17 22:15:41.151897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.151798    2563 scope.go:117] "RemoveContainer" containerID="7e1bc59860e62d03e660e5069e31408a77d331955b41bcda492f3f2c1aea0934"
Jul 17 22:15:41.152063 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:41.152039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf_odh-ai-gateway-infra(ea80ee5b-d505-4db3-a85a-a8e73bbc54e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9"
Jul 17 22:15:41.294743 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.294721    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:41.337723 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.337691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f67b58-ed08-46d8-a7f4-16105605eaba-maas-api-service-ca\") pod \"31f67b58-ed08-46d8-a7f4-16105605eaba\" (UID: \"31f67b58-ed08-46d8-a7f4-16105605eaba\") "
Jul 17 22:15:41.338087 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.338065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f67b58-ed08-46d8-a7f4-16105605eaba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31f67b58-ed08-46d8-a7f4-16105605eaba" (UID: "31f67b58-ed08-46d8-a7f4-16105605eaba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:41.439020 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:41.438988    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31f67b58-ed08-46d8-a7f4-16105605eaba-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:42.158688 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.158659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"
Jul 17 22:15:42.158688 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.158668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w" event={"ID":"31f67b58-ed08-46d8-a7f4-16105605eaba","Type":"ContainerDied","Data":"be6bbf4038b92b7e275f582e9331fc58ed86be62adf5492e361e577a47de0f56"}
Jul 17 22:15:42.159212 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.158737    2563 scope.go:117] "RemoveContainer" containerID="74266718ae75184371925823f4c3b3a7ca26ee9130ed9c21b3568c9fa6243857"
Jul 17 22:15:42.160615 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.160581    2563 scope.go:117] "RemoveContainer" containerID="7e1bc59860e62d03e660e5069e31408a77d331955b41bcda492f3f2c1aea0934"
Jul 17 22:15:42.160859 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:42.160835    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf_odh-ai-gateway-infra(ea80ee5b-d505-4db3-a85a-a8e73bbc54e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9"
Jul 17 22:15:42.193665 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.193620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"]
Jul 17 22:15:42.198530 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.198509    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-gjw2w"]
Jul 17 22:15:42.602687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:42.602652    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" path="/var/lib/kubelet/pods/31f67b58-ed08-46d8-a7f4-16105605eaba/volumes"
Jul 17 22:15:53.597211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:53.597179    2563 scope.go:117] "RemoveContainer" containerID="7e1bc59860e62d03e660e5069e31408a77d331955b41bcda492f3f2c1aea0934"
Jul 17 22:15:54.230400 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:54.230366    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerID="9e0073ddd86148397d239fdffd841623d716edddb85ef7c7830b9e488b972db4" exitCode=6
Jul 17 22:15:54.230576 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:54.230414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" event={"ID":"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9","Type":"ContainerDied","Data":"9e0073ddd86148397d239fdffd841623d716edddb85ef7c7830b9e488b972db4"}
Jul 17 22:15:54.230576 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:54.230464    2563 scope.go:117] "RemoveContainer" containerID="7e1bc59860e62d03e660e5069e31408a77d331955b41bcda492f3f2c1aea0934"
Jul 17 22:15:54.230749 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:54.230730    2563 scope.go:117] "RemoveContainer" containerID="9e0073ddd86148397d239fdffd841623d716edddb85ef7c7830b9e488b972db4"
Jul 17 22:15:54.231011 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:15:54.230987    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf_odh-ai-gateway-infra(ea80ee5b-d505-4db3-a85a-a8e73bbc54e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9"
Jul 17 22:15:55.255291 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:55.255256    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"]
Jul 17 22:15:55.412424 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:55.412397    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:55.459492 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:55.459447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9-maas-api-service-ca\") pod \"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9\" (UID: \"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9\") "
Jul 17 22:15:55.459842 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:55.459819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" (UID: "ea80ee5b-d505-4db3-a85a-a8e73bbc54e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:55.560079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:55.559996    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:15:56.243843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:56.243813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"
Jul 17 22:15:56.243843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:56.243825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf" event={"ID":"ea80ee5b-d505-4db3-a85a-a8e73bbc54e9","Type":"ContainerDied","Data":"7d71bfcf474e286fae2e111e354bd69a6a2c18231bad1d9da6e4a97da8290d0a"}
Jul 17 22:15:56.244073 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:56.243883    2563 scope.go:117] "RemoveContainer" containerID="9e0073ddd86148397d239fdffd841623d716edddb85ef7c7830b9e488b972db4"
Jul 17 22:15:56.266647 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:56.266623    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"]
Jul 17 22:15:56.272309 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:56.272288    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-w69vf"]
Jul 17 22:15:56.602592 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:15:56.602511    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" path="/var/lib/kubelet/pods/ea80ee5b-d505-4db3-a85a-a8e73bbc54e9/volumes"
Jul 17 22:16:05.919238 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919209    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"]
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919563    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919574    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919581    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919586    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919627    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919633    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919645    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919653    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919669    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919726    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919732    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919739    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf41311-f637-45bd-9690-22026dd938c3" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919750    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919756    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919762    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:05.919873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.919767    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:05.924330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.924313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:05.927116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.927084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:16:05.930020 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:05.929996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"]
Jul 17 22:16:06.055901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:06.055863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab0753-953d-4fdf-8881-a777fbd9eb50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82\" (UID: \"8dab0753-953d-4fdf-8881-a777fbd9eb50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:06.156935 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:06.156899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab0753-953d-4fdf-8881-a777fbd9eb50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82\" (UID: \"8dab0753-953d-4fdf-8881-a777fbd9eb50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:06.157534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:06.157513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab0753-953d-4fdf-8881-a777fbd9eb50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82\" (UID: \"8dab0753-953d-4fdf-8881-a777fbd9eb50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:06.237238 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:06.237132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:06.579884 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:06.579856    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"]
Jul 17 22:16:06.581149 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:06.581122    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dab0753_953d_4fdf_8881_a777fbd9eb50.slice/crio-65845da30bbccb7dd283de92fc9a21b88b2123c69817de05ef69c4adebecb486 WatchSource:0}: Error finding container 65845da30bbccb7dd283de92fc9a21b88b2123c69817de05ef69c4adebecb486: Status 404 returned error can't find the container with id 65845da30bbccb7dd283de92fc9a21b88b2123c69817de05ef69c4adebecb486
Jul 17 22:16:07.302405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:07.302371    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerID="3422c7b32bcf7429328ca80459b56e51d9e09ff4dd4b38883e91f7048617023c" exitCode=6
Jul 17 22:16:07.302837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:07.302462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" event={"ID":"8dab0753-953d-4fdf-8881-a777fbd9eb50","Type":"ContainerDied","Data":"3422c7b32bcf7429328ca80459b56e51d9e09ff4dd4b38883e91f7048617023c"}
Jul 17 22:16:07.302837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:07.302556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" event={"ID":"8dab0753-953d-4fdf-8881-a777fbd9eb50","Type":"ContainerStarted","Data":"65845da30bbccb7dd283de92fc9a21b88b2123c69817de05ef69c4adebecb486"}
Jul 17 22:16:07.302837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:07.302633    2563 scope.go:117] "RemoveContainer" containerID="3422c7b32bcf7429328ca80459b56e51d9e09ff4dd4b38883e91f7048617023c"
Jul 17 22:16:08.309733 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:08.309696    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerID="9a15726cb7a77092cf5c7cf69fc4a75068ef51c5df1287789413da62cecc7dc2" exitCode=6
Jul 17 22:16:08.310188 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:08.309750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" event={"ID":"8dab0753-953d-4fdf-8881-a777fbd9eb50","Type":"ContainerDied","Data":"9a15726cb7a77092cf5c7cf69fc4a75068ef51c5df1287789413da62cecc7dc2"}
Jul 17 22:16:08.310188 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:08.309807    2563 scope.go:117] "RemoveContainer" containerID="3422c7b32bcf7429328ca80459b56e51d9e09ff4dd4b38883e91f7048617023c"
Jul 17 22:16:08.310188 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:08.309993    2563 scope.go:117] "RemoveContainer" containerID="9a15726cb7a77092cf5c7cf69fc4a75068ef51c5df1287789413da62cecc7dc2"
Jul 17 22:16:08.310316 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:08.310235    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82_odh-ai-gateway-infra(8dab0753-953d-4fdf-8881-a777fbd9eb50)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50"
Jul 17 22:16:09.316310 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:09.316284    2563 scope.go:117] "RemoveContainer" containerID="9a15726cb7a77092cf5c7cf69fc4a75068ef51c5df1287789413da62cecc7dc2"
Jul 17 22:16:09.316707 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:09.316474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82_odh-ai-gateway-infra(8dab0753-953d-4fdf-8881-a777fbd9eb50)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50"
Jul 17 22:16:10.279655 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.279624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"]
Jul 17 22:16:10.279995 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.279984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:10.280038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.279997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f67b58-ed08-46d8-a7f4-16105605eaba" containerName="revoke-keys"
Jul 17 22:16:10.280038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.280016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:10.280038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.280022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:10.280133 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.280081    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea80ee5b-d505-4db3-a85a-a8e73bbc54e9" containerName="revoke-keys"
Jul 17 22:16:10.280133 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.280091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c02142-9625-40c0-bbc1-7f7d086b6868" containerName="revoke-keys"
Jul 17 22:16:10.284275 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.284259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:10.306967 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.306938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"]
Jul 17 22:16:10.394234 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.394201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5fc835-4940-4b58-b38e-49a453d4ab1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl\" (UID: \"db5fc835-4940-4b58-b38e-49a453d4ab1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:10.494804 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.494770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5fc835-4940-4b58-b38e-49a453d4ab1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl\" (UID: \"db5fc835-4940-4b58-b38e-49a453d4ab1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:10.495416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.495394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5fc835-4940-4b58-b38e-49a453d4ab1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl\" (UID: \"db5fc835-4940-4b58-b38e-49a453d4ab1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:10.594114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.594033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:10.729623 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.729580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"]
Jul 17 22:16:10.731951 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:10.731923    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5fc835_4940_4b58_b38e_49a453d4ab1d.slice/crio-3a2fbe478d08d3a7b0f2c077459a356b9b0cfaf4f99a2e3825b9bbd520ea693e WatchSource:0}: Error finding container 3a2fbe478d08d3a7b0f2c077459a356b9b0cfaf4f99a2e3825b9bbd520ea693e: Status 404 returned error can't find the container with id 3a2fbe478d08d3a7b0f2c077459a356b9b0cfaf4f99a2e3825b9bbd520ea693e
Jul 17 22:16:10.835858 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.835828    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"]
Jul 17 22:16:10.840290 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.840274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:10.851476 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.851449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"]
Jul 17 22:16:10.999765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:10.999732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a406968c-c87e-4a5b-a992-5197e56e1e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95299c-sd7tf\" (UID: \"a406968c-c87e-4a5b-a992-5197e56e1e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:11.100869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.100788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a406968c-c87e-4a5b-a992-5197e56e1e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95299c-sd7tf\" (UID: \"a406968c-c87e-4a5b-a992-5197e56e1e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:11.101380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.101358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a406968c-c87e-4a5b-a992-5197e56e1e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95299c-sd7tf\" (UID: \"a406968c-c87e-4a5b-a992-5197e56e1e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:11.150664 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.150630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:11.285871 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.285842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"]
Jul 17 22:16:11.287528 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:11.287502    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda406968c_c87e_4a5b_a992_5197e56e1e65.slice/crio-95d7ef54691e461106e5e0eeaaa30b0a42eb5d3b6afb3ded3fb49f51a13e7cd2 WatchSource:0}: Error finding container 95d7ef54691e461106e5e0eeaaa30b0a42eb5d3b6afb3ded3fb49f51a13e7cd2: Status 404 returned error can't find the container with id 95d7ef54691e461106e5e0eeaaa30b0a42eb5d3b6afb3ded3fb49f51a13e7cd2
Jul 17 22:16:11.326757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.326730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf" event={"ID":"a406968c-c87e-4a5b-a992-5197e56e1e65","Type":"ContainerStarted","Data":"95d7ef54691e461106e5e0eeaaa30b0a42eb5d3b6afb3ded3fb49f51a13e7cd2"}
Jul 17 22:16:11.328256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.328225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" event={"ID":"db5fc835-4940-4b58-b38e-49a453d4ab1d","Type":"ContainerStarted","Data":"afe63d5e0a9e548e8283ab1a60d269e95b50a8ffb4f6694ceb0b3d4b38a8bf95"}
Jul 17 22:16:11.328358 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.328274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" event={"ID":"db5fc835-4940-4b58-b38e-49a453d4ab1d","Type":"ContainerStarted","Data":"3a2fbe478d08d3a7b0f2c077459a356b9b0cfaf4f99a2e3825b9bbd520ea693e"}
Jul 17 22:16:11.347825 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:11.347783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" podStartSLOduration=1.347769887 podStartE2EDuration="1.347769887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:11.345670454 +0000 UTC m=+2935.257910753" watchObservedRunningTime="2026-07-17 22:16:11.347769887 +0000 UTC m=+2935.260010176"
Jul 17 22:16:12.334636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:12.334539    2563 generic.go:356] "Generic (PLEG): container finished" podID="a406968c-c87e-4a5b-a992-5197e56e1e65" containerID="52e3c3c5c690b0053e911caca5f9a69d7db950dfd7758ba23bd1e23ee27b6382" exitCode=0
Jul 17 22:16:12.334636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:12.334621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf" event={"ID":"a406968c-c87e-4a5b-a992-5197e56e1e65","Type":"ContainerDied","Data":"52e3c3c5c690b0053e911caca5f9a69d7db950dfd7758ba23bd1e23ee27b6382"}
Jul 17 22:16:12.336116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:12.336095    2563 generic.go:356] "Generic (PLEG): container finished" podID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerID="afe63d5e0a9e548e8283ab1a60d269e95b50a8ffb4f6694ceb0b3d4b38a8bf95" exitCode=6
Jul 17 22:16:12.336223 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:12.336126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" event={"ID":"db5fc835-4940-4b58-b38e-49a453d4ab1d","Type":"ContainerDied","Data":"afe63d5e0a9e548e8283ab1a60d269e95b50a8ffb4f6694ceb0b3d4b38a8bf95"}
Jul 17 22:16:12.336345 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:12.336331    2563 scope.go:117] "RemoveContainer" containerID="afe63d5e0a9e548e8283ab1a60d269e95b50a8ffb4f6694ceb0b3d4b38a8bf95"
Jul 17 22:16:13.345924 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.345834    2563 generic.go:356] "Generic (PLEG): container finished" podID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerID="00e806aabd607e2cafcac05dab7ec7befdb88d878f8630dc4b3bd45659ce14f7" exitCode=6
Jul 17 22:16:13.346356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.345926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" event={"ID":"db5fc835-4940-4b58-b38e-49a453d4ab1d","Type":"ContainerDied","Data":"00e806aabd607e2cafcac05dab7ec7befdb88d878f8630dc4b3bd45659ce14f7"}
Jul 17 22:16:13.346356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.345975    2563 scope.go:117] "RemoveContainer" containerID="afe63d5e0a9e548e8283ab1a60d269e95b50a8ffb4f6694ceb0b3d4b38a8bf95"
Jul 17 22:16:13.346356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.346111    2563 scope.go:117] "RemoveContainer" containerID="00e806aabd607e2cafcac05dab7ec7befdb88d878f8630dc4b3bd45659ce14f7"
Jul 17 22:16:13.346356 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:13.346320    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl_odh-ai-gateway-infra(db5fc835-4940-4b58-b38e-49a453d4ab1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d"
Jul 17 22:16:13.405025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.404991    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"]
Jul 17 22:16:13.410540 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.410518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:13.414462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.414435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"]
Jul 17 22:16:13.486138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.486110    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:13.528468 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.528432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34831e8b-934f-475a-aa30-fb94a615809c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl\" (UID: \"34831e8b-934f-475a-aa30-fb94a615809c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:13.629341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.629262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a406968c-c87e-4a5b-a992-5197e56e1e65-maas-api-service-ca\") pod \"a406968c-c87e-4a5b-a992-5197e56e1e65\" (UID: \"a406968c-c87e-4a5b-a992-5197e56e1e65\") "
Jul 17 22:16:13.629488 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.629398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34831e8b-934f-475a-aa30-fb94a615809c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl\" (UID: \"34831e8b-934f-475a-aa30-fb94a615809c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:13.629636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.629592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a406968c-c87e-4a5b-a992-5197e56e1e65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a406968c-c87e-4a5b-a992-5197e56e1e65" (UID: "a406968c-c87e-4a5b-a992-5197e56e1e65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:13.630003 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.629987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34831e8b-934f-475a-aa30-fb94a615809c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl\" (UID: \"34831e8b-934f-475a-aa30-fb94a615809c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:13.725486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.725441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:13.730881 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.730859    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a406968c-c87e-4a5b-a992-5197e56e1e65-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:16:13.866279 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:13.866236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"]
Jul 17 22:16:13.866393 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:13.866346    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34831e8b_934f_475a_aa30_fb94a615809c.slice/crio-72b86194cb5fe591280e920f8aba8cf024407293c95651272bc8e912d61486df WatchSource:0}: Error finding container 72b86194cb5fe591280e920f8aba8cf024407293c95651272bc8e912d61486df: Status 404 returned error can't find the container with id 72b86194cb5fe591280e920f8aba8cf024407293c95651272bc8e912d61486df
Jul 17 22:16:14.353784 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.353740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" event={"ID":"34831e8b-934f-475a-aa30-fb94a615809c","Type":"ContainerStarted","Data":"9e25500e513f12199b3f43b16f6920c9852fdce39388686ac9f4f3fd7c53aba0"}
Jul 17 22:16:14.354239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.353804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" event={"ID":"34831e8b-934f-475a-aa30-fb94a615809c","Type":"ContainerStarted","Data":"72b86194cb5fe591280e920f8aba8cf024407293c95651272bc8e912d61486df"}
Jul 17 22:16:14.355359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.355334    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf"
Jul 17 22:16:14.355359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.355339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95299c-sd7tf" event={"ID":"a406968c-c87e-4a5b-a992-5197e56e1e65","Type":"ContainerDied","Data":"95d7ef54691e461106e5e0eeaaa30b0a42eb5d3b6afb3ded3fb49f51a13e7cd2"}
Jul 17 22:16:14.355533 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.355389    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d7ef54691e461106e5e0eeaaa30b0a42eb5d3b6afb3ded3fb49f51a13e7cd2"
Jul 17 22:16:14.357222 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.357204    2563 scope.go:117] "RemoveContainer" containerID="00e806aabd607e2cafcac05dab7ec7befdb88d878f8630dc4b3bd45659ce14f7"
Jul 17 22:16:14.357409 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:14.357389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl_odh-ai-gateway-infra(db5fc835-4940-4b58-b38e-49a453d4ab1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d"
Jul 17 22:16:14.370770 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:14.370727    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" podStartSLOduration=1.37071552 podStartE2EDuration="1.37071552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:14.368723451 +0000 UTC m=+2938.280963740" watchObservedRunningTime="2026-07-17 22:16:14.37071552 +0000 UTC m=+2938.282955816"
Jul 17 22:16:15.364499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:15.364463    2563 generic.go:356] "Generic (PLEG): container finished" podID="34831e8b-934f-475a-aa30-fb94a615809c" containerID="9e25500e513f12199b3f43b16f6920c9852fdce39388686ac9f4f3fd7c53aba0" exitCode=6
Jul 17 22:16:15.364914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:15.364556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" event={"ID":"34831e8b-934f-475a-aa30-fb94a615809c","Type":"ContainerDied","Data":"9e25500e513f12199b3f43b16f6920c9852fdce39388686ac9f4f3fd7c53aba0"}
Jul 17 22:16:15.364914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:15.364740    2563 scope.go:117] "RemoveContainer" containerID="9e25500e513f12199b3f43b16f6920c9852fdce39388686ac9f4f3fd7c53aba0"
Jul 17 22:16:16.372110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:16.372025    2563 generic.go:356] "Generic (PLEG): container finished" podID="34831e8b-934f-475a-aa30-fb94a615809c" containerID="64dbbd77cb6fa74a381d45a50eb53b3f16390cc7dbd804185fd785b929b25a91" exitCode=6
Jul 17 22:16:16.372110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:16.372078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" event={"ID":"34831e8b-934f-475a-aa30-fb94a615809c","Type":"ContainerDied","Data":"64dbbd77cb6fa74a381d45a50eb53b3f16390cc7dbd804185fd785b929b25a91"}
Jul 17 22:16:16.372571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:16.372133    2563 scope.go:117] "RemoveContainer" containerID="9e25500e513f12199b3f43b16f6920c9852fdce39388686ac9f4f3fd7c53aba0"
Jul 17 22:16:16.372571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:16.372286    2563 scope.go:117] "RemoveContainer" containerID="64dbbd77cb6fa74a381d45a50eb53b3f16390cc7dbd804185fd785b929b25a91"
Jul 17 22:16:16.372571 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:16.372507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl_odh-ai-gateway-infra(34831e8b-934f-475a-aa30-fb94a615809c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" podUID="34831e8b-934f-475a-aa30-fb94a615809c"
Jul 17 22:16:17.379836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:17.379763    2563 scope.go:117] "RemoveContainer" containerID="64dbbd77cb6fa74a381d45a50eb53b3f16390cc7dbd804185fd785b929b25a91"
Jul 17 22:16:17.380186 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:17.379967    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl_odh-ai-gateway-infra(34831e8b-934f-475a-aa30-fb94a615809c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" podUID="34831e8b-934f-475a-aa30-fb94a615809c"
Jul 17 22:16:20.597166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:20.597136    2563 scope.go:117] "RemoveContainer" containerID="9a15726cb7a77092cf5c7cf69fc4a75068ef51c5df1287789413da62cecc7dc2"
Jul 17 22:16:21.403212 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:21.403179    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerID="c37f75249c2de6840586dcc69ff6644cf25670dab5b405ee04163341ed4528b9" exitCode=6
Jul 17 22:16:21.403388 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:21.403255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" event={"ID":"8dab0753-953d-4fdf-8881-a777fbd9eb50","Type":"ContainerDied","Data":"c37f75249c2de6840586dcc69ff6644cf25670dab5b405ee04163341ed4528b9"}
Jul 17 22:16:21.403388 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:21.403342    2563 scope.go:117] "RemoveContainer" containerID="9a15726cb7a77092cf5c7cf69fc4a75068ef51c5df1287789413da62cecc7dc2"
Jul 17 22:16:21.403659 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:21.403638    2563 scope.go:117] "RemoveContainer" containerID="c37f75249c2de6840586dcc69ff6644cf25670dab5b405ee04163341ed4528b9"
Jul 17 22:16:21.403857 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:21.403838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82_odh-ai-gateway-infra(8dab0753-953d-4fdf-8881-a777fbd9eb50)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50"
Jul 17 22:16:22.435023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:22.434990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"]
Jul 17 22:16:22.577444 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:22.577419    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:22.605716 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:22.605688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab0753-953d-4fdf-8881-a777fbd9eb50-maas-api-service-ca\") pod \"8dab0753-953d-4fdf-8881-a777fbd9eb50\" (UID: \"8dab0753-953d-4fdf-8881-a777fbd9eb50\") "
Jul 17 22:16:22.606041 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:22.606017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dab0753-953d-4fdf-8881-a777fbd9eb50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8dab0753-953d-4fdf-8881-a777fbd9eb50" (UID: "8dab0753-953d-4fdf-8881-a777fbd9eb50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:22.707015 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:22.706934    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab0753-953d-4fdf-8881-a777fbd9eb50-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:16:23.417233 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:23.417204    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"
Jul 17 22:16:23.417233 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:23.417228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82" event={"ID":"8dab0753-953d-4fdf-8881-a777fbd9eb50","Type":"ContainerDied","Data":"65845da30bbccb7dd283de92fc9a21b88b2123c69817de05ef69c4adebecb486"}
Jul 17 22:16:23.417459 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:23.417283    2563 scope.go:117] "RemoveContainer" containerID="c37f75249c2de6840586dcc69ff6644cf25670dab5b405ee04163341ed4528b9"
Jul 17 22:16:23.437027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:23.437001    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"]
Jul 17 22:16:23.440692 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:23.440671    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-9kq82"]
Jul 17 22:16:24.597020 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:24.596993    2563 scope.go:117] "RemoveContainer" containerID="00e806aabd607e2cafcac05dab7ec7befdb88d878f8630dc4b3bd45659ce14f7"
Jul 17 22:16:24.603263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:24.603163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" path="/var/lib/kubelet/pods/8dab0753-953d-4fdf-8881-a777fbd9eb50/volumes"
Jul 17 22:16:25.431383 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:25.431344    2563 generic.go:356] "Generic (PLEG): container finished" podID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerID="a41b4ef28cf8403cc20ae85865a9e1ef1bb519234c4ad6a8e29dc0e631d57b64" exitCode=6
Jul 17 22:16:25.431559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:25.431412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" event={"ID":"db5fc835-4940-4b58-b38e-49a453d4ab1d","Type":"ContainerDied","Data":"a41b4ef28cf8403cc20ae85865a9e1ef1bb519234c4ad6a8e29dc0e631d57b64"}
Jul 17 22:16:25.431559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:25.431504    2563 scope.go:117] "RemoveContainer" containerID="00e806aabd607e2cafcac05dab7ec7befdb88d878f8630dc4b3bd45659ce14f7"
Jul 17 22:16:25.431852 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:25.431830    2563 scope.go:117] "RemoveContainer" containerID="a41b4ef28cf8403cc20ae85865a9e1ef1bb519234c4ad6a8e29dc0e631d57b64"
Jul 17 22:16:25.432098 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:25.432071    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl_odh-ai-gateway-infra(db5fc835-4940-4b58-b38e-49a453d4ab1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d"
Jul 17 22:16:26.455729 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:26.455695    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"]
Jul 17 22:16:26.600021 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:26.599993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:26.635965 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:26.635931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5fc835-4940-4b58-b38e-49a453d4ab1d-maas-api-service-ca\") pod \"db5fc835-4940-4b58-b38e-49a453d4ab1d\" (UID: \"db5fc835-4940-4b58-b38e-49a453d4ab1d\") "
Jul 17 22:16:26.636378 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:26.636348    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5fc835-4940-4b58-b38e-49a453d4ab1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db5fc835-4940-4b58-b38e-49a453d4ab1d" (UID: "db5fc835-4940-4b58-b38e-49a453d4ab1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:26.737064 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:26.736982    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5fc835-4940-4b58-b38e-49a453d4ab1d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:16:27.444726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:27.444697    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"
Jul 17 22:16:27.444726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:27.444713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl" event={"ID":"db5fc835-4940-4b58-b38e-49a453d4ab1d","Type":"ContainerDied","Data":"3a2fbe478d08d3a7b0f2c077459a356b9b0cfaf4f99a2e3825b9bbd520ea693e"}
Jul 17 22:16:27.444970 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:27.444768    2563 scope.go:117] "RemoveContainer" containerID="a41b4ef28cf8403cc20ae85865a9e1ef1bb519234c4ad6a8e29dc0e631d57b64"
Jul 17 22:16:27.468301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:27.468278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"]
Jul 17 22:16:27.471824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:27.471803    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-sthvl"]
Jul 17 22:16:28.051944 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.051911    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"]
Jul 17 22:16:28.052297 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a406968c-c87e-4a5b-a992-5197e56e1e65" containerName="revoke-keys"
Jul 17 22:16:28.052346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052299    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a406968c-c87e-4a5b-a992-5197e56e1e65" containerName="revoke-keys"
Jul 17 22:16:28.052346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052314    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052335    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052346    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052358    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052363    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052376    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052438    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a406968c-c87e-4a5b-a992-5197e56e1e65" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052445    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052453    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052459    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052465    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052471    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:28.052535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.052477    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dab0753-953d-4fdf-8881-a777fbd9eb50" containerName="revoke-keys"
Jul 17 22:16:28.056591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.056571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:28.061622 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.061529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"]
Jul 17 22:16:28.147861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.147826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81265f84-d3d4-440f-9daa-39c49288d1e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8\" (UID: \"81265f84-d3d4-440f-9daa-39c49288d1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:28.248938 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.248909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81265f84-d3d4-440f-9daa-39c49288d1e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8\" (UID: \"81265f84-d3d4-440f-9daa-39c49288d1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:28.249512 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.249492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81265f84-d3d4-440f-9daa-39c49288d1e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8\" (UID: \"81265f84-d3d4-440f-9daa-39c49288d1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:28.368728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.368632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:28.502666 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.502635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"]
Jul 17 22:16:28.504277 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:28.504255    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81265f84_d3d4_440f_9daa_39c49288d1e0.slice/crio-ebe87a070925d82d4d974b5966fa99c4569ecdc7bf27186cf6013d6dc0571655 WatchSource:0}: Error finding container ebe87a070925d82d4d974b5966fa99c4569ecdc7bf27186cf6013d6dc0571655: Status 404 returned error can't find the container with id ebe87a070925d82d4d974b5966fa99c4569ecdc7bf27186cf6013d6dc0571655
Jul 17 22:16:28.602327 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:28.602298    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" path="/var/lib/kubelet/pods/db5fc835-4940-4b58-b38e-49a453d4ab1d/volumes"
Jul 17 22:16:29.456791 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:29.456759    2563 generic.go:356] "Generic (PLEG): container finished" podID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerID="c0552bda8705857f643d9b40e3a98b7ea2f8a5d381d841d6bfe58e0da2d357c4" exitCode=6
Jul 17 22:16:29.456963 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:29.456846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" event={"ID":"81265f84-d3d4-440f-9daa-39c49288d1e0","Type":"ContainerDied","Data":"c0552bda8705857f643d9b40e3a98b7ea2f8a5d381d841d6bfe58e0da2d357c4"}
Jul 17 22:16:29.456963 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:29.456905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" event={"ID":"81265f84-d3d4-440f-9daa-39c49288d1e0","Type":"ContainerStarted","Data":"ebe87a070925d82d4d974b5966fa99c4569ecdc7bf27186cf6013d6dc0571655"}
Jul 17 22:16:29.457039 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:29.457011    2563 scope.go:117] "RemoveContainer" containerID="c0552bda8705857f643d9b40e3a98b7ea2f8a5d381d841d6bfe58e0da2d357c4"
Jul 17 22:16:29.597319 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:29.597284    2563 scope.go:117] "RemoveContainer" containerID="64dbbd77cb6fa74a381d45a50eb53b3f16390cc7dbd804185fd785b929b25a91"
Jul 17 22:16:30.466345 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.466306    2563 generic.go:356] "Generic (PLEG): container finished" podID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerID="62795d6a4ae85b326805b2ee7dd7e5198a87f587092f6ccfa4a3d5018cee47fd" exitCode=6
Jul 17 22:16:30.466534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.466388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" event={"ID":"81265f84-d3d4-440f-9daa-39c49288d1e0","Type":"ContainerDied","Data":"62795d6a4ae85b326805b2ee7dd7e5198a87f587092f6ccfa4a3d5018cee47fd"}
Jul 17 22:16:30.466534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.466480    2563 scope.go:117] "RemoveContainer" containerID="c0552bda8705857f643d9b40e3a98b7ea2f8a5d381d841d6bfe58e0da2d357c4"
Jul 17 22:16:30.466717 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.466702    2563 scope.go:117] "RemoveContainer" containerID="62795d6a4ae85b326805b2ee7dd7e5198a87f587092f6ccfa4a3d5018cee47fd"
Jul 17 22:16:30.466936 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:30.466911    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8_odh-ai-gateway-infra(81265f84-d3d4-440f-9daa-39c49288d1e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0"
Jul 17 22:16:30.468338 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.468316    2563 generic.go:356] "Generic (PLEG): container finished" podID="34831e8b-934f-475a-aa30-fb94a615809c" containerID="909e9544de7e39c06e9135a8da02a278a41368a7dbc7fdda91b44cf6f5e3cfa5" exitCode=6
Jul 17 22:16:30.468429 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.468354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" event={"ID":"34831e8b-934f-475a-aa30-fb94a615809c","Type":"ContainerDied","Data":"909e9544de7e39c06e9135a8da02a278a41368a7dbc7fdda91b44cf6f5e3cfa5"}
Jul 17 22:16:30.468619 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.468584    2563 scope.go:117] "RemoveContainer" containerID="909e9544de7e39c06e9135a8da02a278a41368a7dbc7fdda91b44cf6f5e3cfa5"
Jul 17 22:16:30.468777 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:30.468760    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl_odh-ai-gateway-infra(34831e8b-934f-475a-aa30-fb94a615809c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" podUID="34831e8b-934f-475a-aa30-fb94a615809c"
Jul 17 22:16:30.477680 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:30.477663    2563 scope.go:117] "RemoveContainer" containerID="64dbbd77cb6fa74a381d45a50eb53b3f16390cc7dbd804185fd785b929b25a91"
Jul 17 22:16:31.475736 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:31.475704    2563 scope.go:117] "RemoveContainer" containerID="62795d6a4ae85b326805b2ee7dd7e5198a87f587092f6ccfa4a3d5018cee47fd"
Jul 17 22:16:31.476175 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:31.475930    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8_odh-ai-gateway-infra(81265f84-d3d4-440f-9daa-39c49288d1e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0"
Jul 17 22:16:31.502450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:31.502421    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"]
Jul 17 22:16:31.649736 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:31.649706    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:31.676854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:31.676827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34831e8b-934f-475a-aa30-fb94a615809c-maas-api-service-ca\") pod \"34831e8b-934f-475a-aa30-fb94a615809c\" (UID: \"34831e8b-934f-475a-aa30-fb94a615809c\") "
Jul 17 22:16:31.677462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:31.677440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34831e8b-934f-475a-aa30-fb94a615809c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34831e8b-934f-475a-aa30-fb94a615809c" (UID: "34831e8b-934f-475a-aa30-fb94a615809c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:31.778022 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:31.777937    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34831e8b-934f-475a-aa30-fb94a615809c-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:16:32.482870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:32.482835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl" event={"ID":"34831e8b-934f-475a-aa30-fb94a615809c","Type":"ContainerDied","Data":"72b86194cb5fe591280e920f8aba8cf024407293c95651272bc8e912d61486df"}
Jul 17 22:16:32.482870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:32.482855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"
Jul 17 22:16:32.483395 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:32.482895    2563 scope.go:117] "RemoveContainer" containerID="909e9544de7e39c06e9135a8da02a278a41368a7dbc7fdda91b44cf6f5e3cfa5"
Jul 17 22:16:32.506488 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:32.506457    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"]
Jul 17 22:16:32.511772 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:32.511750    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6qrdl"]
Jul 17 22:16:32.603162 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:32.603128    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34831e8b-934f-475a-aa30-fb94a615809c" path="/var/lib/kubelet/pods/34831e8b-934f-475a-aa30-fb94a615809c/volumes"
Jul 17 22:16:36.906903 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.906868    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"]
Jul 17 22:16:36.907277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907220    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:36.907277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907231    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:36.907277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907244    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:36.907277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907249    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5fc835-4940-4b58-b38e-49a453d4ab1d" containerName="revoke-keys"
Jul 17 22:16:36.907277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907267    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:36.907277 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907273    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:36.907490 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907328    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:36.907490 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.907336    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:36.911931 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.911915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:36.918417 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:36.918389    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"]
Jul 17 22:16:37.022109 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:37.022071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c646af7a-6101-468a-b006-f44bd04a3582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8\" (UID: \"c646af7a-6101-468a-b006-f44bd04a3582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:37.122721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:37.122686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c646af7a-6101-468a-b006-f44bd04a3582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8\" (UID: \"c646af7a-6101-468a-b006-f44bd04a3582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:37.123268 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:37.123250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c646af7a-6101-468a-b006-f44bd04a3582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8\" (UID: \"c646af7a-6101-468a-b006-f44bd04a3582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:37.223908 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:37.223874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:37.368182 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:37.368156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"]
Jul 17 22:16:37.370439 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:37.370412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc646af7a_6101_468a_b006_f44bd04a3582.slice/crio-d9bbb9ca864edacd7ffbeb70f823b10793e94b1eaa81d789c1103f4b68dde0f9 WatchSource:0}: Error finding container d9bbb9ca864edacd7ffbeb70f823b10793e94b1eaa81d789c1103f4b68dde0f9: Status 404 returned error can't find the container with id d9bbb9ca864edacd7ffbeb70f823b10793e94b1eaa81d789c1103f4b68dde0f9
Jul 17 22:16:37.508523 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:37.508437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8" event={"ID":"c646af7a-6101-468a-b006-f44bd04a3582","Type":"ContainerStarted","Data":"d9bbb9ca864edacd7ffbeb70f823b10793e94b1eaa81d789c1103f4b68dde0f9"}
Jul 17 22:16:38.514794 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:38.514759    2563 generic.go:356] "Generic (PLEG): container finished" podID="c646af7a-6101-468a-b006-f44bd04a3582" containerID="91ed208f57dbfa203e8a9c002fe0886b65be1574e38e3d0733f09a7b0c97a77e" exitCode=0
Jul 17 22:16:38.515211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:38.514849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8" event={"ID":"c646af7a-6101-468a-b006-f44bd04a3582","Type":"ContainerDied","Data":"91ed208f57dbfa203e8a9c002fe0886b65be1574e38e3d0733f09a7b0c97a77e"}
Jul 17 22:16:39.654444 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:39.654412    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:39.745358 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:39.745321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c646af7a-6101-468a-b006-f44bd04a3582-maas-api-service-ca\") pod \"c646af7a-6101-468a-b006-f44bd04a3582\" (UID: \"c646af7a-6101-468a-b006-f44bd04a3582\") "
Jul 17 22:16:39.745704 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:39.745683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c646af7a-6101-468a-b006-f44bd04a3582-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c646af7a-6101-468a-b006-f44bd04a3582" (UID: "c646af7a-6101-468a-b006-f44bd04a3582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:39.846232 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:39.846152    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c646af7a-6101-468a-b006-f44bd04a3582-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:16:40.527364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:40.527336    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8"
Jul 17 22:16:40.527546 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:40.527366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8" event={"ID":"c646af7a-6101-468a-b006-f44bd04a3582","Type":"ContainerDied","Data":"d9bbb9ca864edacd7ffbeb70f823b10793e94b1eaa81d789c1103f4b68dde0f9"}
Jul 17 22:16:40.527546 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:40.527412    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9bbb9ca864edacd7ffbeb70f823b10793e94b1eaa81d789c1103f4b68dde0f9"
Jul 17 22:16:42.597405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:42.597376    2563 scope.go:117] "RemoveContainer" containerID="62795d6a4ae85b326805b2ee7dd7e5198a87f587092f6ccfa4a3d5018cee47fd"
Jul 17 22:16:43.545021 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:43.544983    2563 generic.go:356] "Generic (PLEG): container finished" podID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerID="d2fcf6ce4bf74068e5c3328abea418e1eaaae902a235e6df873be48e911c4656" exitCode=6
Jul 17 22:16:43.545174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:43.545062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" event={"ID":"81265f84-d3d4-440f-9daa-39c49288d1e0","Type":"ContainerDied","Data":"d2fcf6ce4bf74068e5c3328abea418e1eaaae902a235e6df873be48e911c4656"}
Jul 17 22:16:43.545174 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:43.545128    2563 scope.go:117] "RemoveContainer" containerID="62795d6a4ae85b326805b2ee7dd7e5198a87f587092f6ccfa4a3d5018cee47fd"
Jul 17 22:16:43.545510 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:43.545496    2563 scope.go:117] "RemoveContainer" containerID="d2fcf6ce4bf74068e5c3328abea418e1eaaae902a235e6df873be48e911c4656"
Jul 17 22:16:43.545727 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:43.545702    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8_odh-ai-gateway-infra(81265f84-d3d4-440f-9daa-39c49288d1e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0"
Jul 17 22:16:44.570378 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:44.570343    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"]
Jul 17 22:16:44.707826 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:44.707799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:44.784934 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:44.784896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81265f84-d3d4-440f-9daa-39c49288d1e0-maas-api-service-ca\") pod \"81265f84-d3d4-440f-9daa-39c49288d1e0\" (UID: \"81265f84-d3d4-440f-9daa-39c49288d1e0\") "
Jul 17 22:16:44.785274 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:44.785249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81265f84-d3d4-440f-9daa-39c49288d1e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81265f84-d3d4-440f-9daa-39c49288d1e0" (UID: "81265f84-d3d4-440f-9daa-39c49288d1e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:44.886083 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:44.886011    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81265f84-d3d4-440f-9daa-39c49288d1e0-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:16:45.560096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:45.560068    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"
Jul 17 22:16:45.560263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:45.560107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8" event={"ID":"81265f84-d3d4-440f-9daa-39c49288d1e0","Type":"ContainerDied","Data":"ebe87a070925d82d4d974b5966fa99c4569ecdc7bf27186cf6013d6dc0571655"}
Jul 17 22:16:45.560263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:45.560167    2563 scope.go:117] "RemoveContainer" containerID="d2fcf6ce4bf74068e5c3328abea418e1eaaae902a235e6df873be48e911c4656"
Jul 17 22:16:45.582521 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:45.582495    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"]
Jul 17 22:16:45.588147 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:45.588122    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bbvd8"]
Jul 17 22:16:46.603165 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:46.603134    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" path="/var/lib/kubelet/pods/81265f84-d3d4-440f-9daa-39c49288d1e0/volumes"
Jul 17 22:16:55.959678 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.959647    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"]
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960024    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960038    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960045    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960051    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c646af7a-6101-468a-b006-f44bd04a3582" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960063    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c646af7a-6101-468a-b006-f44bd04a3582" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960071    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960077    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960149    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960157    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34831e8b-934f-475a-aa30-fb94a615809c" containerName="revoke-keys"
Jul 17 22:16:55.960457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960164    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:16:55.960457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.960170    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c646af7a-6101-468a-b006-f44bd04a3582" containerName="revoke-keys"
Jul 17 22:16:55.963189 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.963172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:16:55.965486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.965468    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:16:55.975256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:55.975225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"]
Jul 17 22:16:56.083349 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:56.083310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c842ed7-43de-44f2-a748-f633affba9a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh\" (UID: \"6c842ed7-43de-44f2-a748-f633affba9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:16:56.184846 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:56.184808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c842ed7-43de-44f2-a748-f633affba9a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh\" (UID: \"6c842ed7-43de-44f2-a748-f633affba9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:16:56.185436 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:56.185413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c842ed7-43de-44f2-a748-f633affba9a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh\" (UID: \"6c842ed7-43de-44f2-a748-f633affba9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:16:56.280050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:56.279972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:16:56.437457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:56.437432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"]
Jul 17 22:16:56.439888 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:16:56.439862    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c842ed7_43de_44f2_a748_f633affba9a2.slice/crio-1b7e6024d07e65392244ad7b53f755ccc49779d484f0a5e859ea2ea9b48c9c5c WatchSource:0}: Error finding container 1b7e6024d07e65392244ad7b53f755ccc49779d484f0a5e859ea2ea9b48c9c5c: Status 404 returned error can't find the container with id 1b7e6024d07e65392244ad7b53f755ccc49779d484f0a5e859ea2ea9b48c9c5c
Jul 17 22:16:56.624521 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:56.624439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" event={"ID":"6c842ed7-43de-44f2-a748-f633affba9a2","Type":"ContainerStarted","Data":"1b7e6024d07e65392244ad7b53f755ccc49779d484f0a5e859ea2ea9b48c9c5c"}
Jul 17 22:16:57.630565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:57.630535    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c842ed7-43de-44f2-a748-f633affba9a2" containerID="50c3852d1381cfd8b18ed75ea968016182a498f2889aa5952563d2deb926fa8a" exitCode=6
Jul 17 22:16:57.630972 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:57.630637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" event={"ID":"6c842ed7-43de-44f2-a748-f633affba9a2","Type":"ContainerDied","Data":"50c3852d1381cfd8b18ed75ea968016182a498f2889aa5952563d2deb926fa8a"}
Jul 17 22:16:57.630972 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:57.630844    2563 scope.go:117] "RemoveContainer" containerID="50c3852d1381cfd8b18ed75ea968016182a498f2889aa5952563d2deb926fa8a"
Jul 17 22:16:58.637980 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:58.637893    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c842ed7-43de-44f2-a748-f633affba9a2" containerID="a6529287c0e221a98bb8dff71e953ee75ca1fd3495693434642fa36355ab7cd8" exitCode=6
Jul 17 22:16:58.638392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:58.637977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" event={"ID":"6c842ed7-43de-44f2-a748-f633affba9a2","Type":"ContainerDied","Data":"a6529287c0e221a98bb8dff71e953ee75ca1fd3495693434642fa36355ab7cd8"}
Jul 17 22:16:58.638392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:58.638034    2563 scope.go:117] "RemoveContainer" containerID="50c3852d1381cfd8b18ed75ea968016182a498f2889aa5952563d2deb926fa8a"
Jul 17 22:16:58.638392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:58.638153    2563 scope.go:117] "RemoveContainer" containerID="a6529287c0e221a98bb8dff71e953ee75ca1fd3495693434642fa36355ab7cd8"
Jul 17 22:16:58.638392 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:58.638354    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh_odh-ai-gateway-infra(6c842ed7-43de-44f2-a748-f633affba9a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" podUID="6c842ed7-43de-44f2-a748-f633affba9a2"
Jul 17 22:16:59.644820 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:16:59.644792    2563 scope.go:117] "RemoveContainer" containerID="a6529287c0e221a98bb8dff71e953ee75ca1fd3495693434642fa36355ab7cd8"
Jul 17 22:16:59.645195 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:16:59.644978    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh_odh-ai-gateway-infra(6c842ed7-43de-44f2-a748-f633affba9a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" podUID="6c842ed7-43de-44f2-a748-f633affba9a2"
Jul 17 22:17:00.287200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.287159    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"]
Jul 17 22:17:00.287640 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.287627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81265f84-d3d4-440f-9daa-39c49288d1e0" containerName="revoke-keys"
Jul 17 22:17:00.291873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.291858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:00.297131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.297103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"]
Jul 17 22:17:00.422327 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.422288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f929c389-f747-4432-a998-e3197fb143ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl\" (UID: \"f929c389-f747-4432-a998-e3197fb143ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:00.522918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.522882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f929c389-f747-4432-a998-e3197fb143ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl\" (UID: \"f929c389-f747-4432-a998-e3197fb143ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:00.523505 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.523485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f929c389-f747-4432-a998-e3197fb143ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl\" (UID: \"f929c389-f747-4432-a998-e3197fb143ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:00.603960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.603882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:00.742161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:00.742134    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"]
Jul 17 22:17:00.745157 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:00.745124    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf929c389_f747_4432_a998_e3197fb143ca.slice/crio-59977144ea03d66ad11b0998ac90a4f30e35d835788a767b6989588cc83850f5 WatchSource:0}: Error finding container 59977144ea03d66ad11b0998ac90a4f30e35d835788a767b6989588cc83850f5: Status 404 returned error can't find the container with id 59977144ea03d66ad11b0998ac90a4f30e35d835788a767b6989588cc83850f5
Jul 17 22:17:01.657163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:01.657073    2563 generic.go:356] "Generic (PLEG): container finished" podID="f929c389-f747-4432-a998-e3197fb143ca" containerID="81f5d87fc244aa3f415ef31e9ac8a43fb9ef7968022acdb748461e869fb7b035" exitCode=6
Jul 17 22:17:01.657315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:01.657164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" event={"ID":"f929c389-f747-4432-a998-e3197fb143ca","Type":"ContainerDied","Data":"81f5d87fc244aa3f415ef31e9ac8a43fb9ef7968022acdb748461e869fb7b035"}
Jul 17 22:17:01.657315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:01.657219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" event={"ID":"f929c389-f747-4432-a998-e3197fb143ca","Type":"ContainerStarted","Data":"59977144ea03d66ad11b0998ac90a4f30e35d835788a767b6989588cc83850f5"}
Jul 17 22:17:01.657401 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:01.657317    2563 scope.go:117] "RemoveContainer" containerID="81f5d87fc244aa3f415ef31e9ac8a43fb9ef7968022acdb748461e869fb7b035"
Jul 17 22:17:02.664752 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:02.664717    2563 generic.go:356] "Generic (PLEG): container finished" podID="f929c389-f747-4432-a998-e3197fb143ca" containerID="154df00a6b6b5ef86b2ab3750d59ca904efe7a37019c4ffd42b349b40a161d02" exitCode=6
Jul 17 22:17:02.665170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:02.664801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" event={"ID":"f929c389-f747-4432-a998-e3197fb143ca","Type":"ContainerDied","Data":"154df00a6b6b5ef86b2ab3750d59ca904efe7a37019c4ffd42b349b40a161d02"}
Jul 17 22:17:02.665170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:02.664879    2563 scope.go:117] "RemoveContainer" containerID="81f5d87fc244aa3f415ef31e9ac8a43fb9ef7968022acdb748461e869fb7b035"
Jul 17 22:17:02.665170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:02.665059    2563 scope.go:117] "RemoveContainer" containerID="154df00a6b6b5ef86b2ab3750d59ca904efe7a37019c4ffd42b349b40a161d02"
Jul 17 22:17:02.665358 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:02.665325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl_odh-ai-gateway-infra(f929c389-f747-4432-a998-e3197fb143ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" podUID="f929c389-f747-4432-a998-e3197fb143ca"
Jul 17 22:17:03.607786 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.607749    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"]
Jul 17 22:17:03.612634 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.612582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:03.618232 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.618198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"]
Jul 17 22:17:03.672551 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.672519    2563 scope.go:117] "RemoveContainer" containerID="154df00a6b6b5ef86b2ab3750d59ca904efe7a37019c4ffd42b349b40a161d02"
Jul 17 22:17:03.672919 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:03.672764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl_odh-ai-gateway-infra(f929c389-f747-4432-a998-e3197fb143ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" podUID="f929c389-f747-4432-a998-e3197fb143ca"
Jul 17 22:17:03.749577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.749540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c778bdf-80c8-4bfc-ae54-cd894be6205d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x\" (UID: \"8c778bdf-80c8-4bfc-ae54-cd894be6205d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:03.850854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.850823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c778bdf-80c8-4bfc-ae54-cd894be6205d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x\" (UID: \"8c778bdf-80c8-4bfc-ae54-cd894be6205d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:03.851409 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.851391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c778bdf-80c8-4bfc-ae54-cd894be6205d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x\" (UID: \"8c778bdf-80c8-4bfc-ae54-cd894be6205d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:03.925982 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:03.925956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:04.066950 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:04.066920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"]
Jul 17 22:17:04.068404 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:04.068367    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c778bdf_80c8_4bfc_ae54_cd894be6205d.slice/crio-f197196feeb9a3f7ff6e00815769cca85495b2b9bc2d8847c492d0837c78c8ad WatchSource:0}: Error finding container f197196feeb9a3f7ff6e00815769cca85495b2b9bc2d8847c492d0837c78c8ad: Status 404 returned error can't find the container with id f197196feeb9a3f7ff6e00815769cca85495b2b9bc2d8847c492d0837c78c8ad
Jul 17 22:17:04.681380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:04.681344    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerID="6f9ccb4df3cf6c8acc7ea5f3c85e7025e13e33ba540b7bb1ba09991e7bd3e5fc" exitCode=6
Jul 17 22:17:04.681795 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:04.681387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" event={"ID":"8c778bdf-80c8-4bfc-ae54-cd894be6205d","Type":"ContainerDied","Data":"6f9ccb4df3cf6c8acc7ea5f3c85e7025e13e33ba540b7bb1ba09991e7bd3e5fc"}
Jul 17 22:17:04.681795 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:04.681434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" event={"ID":"8c778bdf-80c8-4bfc-ae54-cd894be6205d","Type":"ContainerStarted","Data":"f197196feeb9a3f7ff6e00815769cca85495b2b9bc2d8847c492d0837c78c8ad"}
Jul 17 22:17:04.681795 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:04.681640    2563 scope.go:117] "RemoveContainer" containerID="6f9ccb4df3cf6c8acc7ea5f3c85e7025e13e33ba540b7bb1ba09991e7bd3e5fc"
Jul 17 22:17:05.688779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:05.688746    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerID="e23cbb47dc71cf96f87b4cdac8e8cf6c7805376abdffb3b3f0d481b050bb7107" exitCode=6
Jul 17 22:17:05.689200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:05.688782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" event={"ID":"8c778bdf-80c8-4bfc-ae54-cd894be6205d","Type":"ContainerDied","Data":"e23cbb47dc71cf96f87b4cdac8e8cf6c7805376abdffb3b3f0d481b050bb7107"}
Jul 17 22:17:05.689200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:05.688857    2563 scope.go:117] "RemoveContainer" containerID="6f9ccb4df3cf6c8acc7ea5f3c85e7025e13e33ba540b7bb1ba09991e7bd3e5fc"
Jul 17 22:17:05.689200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:05.689046    2563 scope.go:117] "RemoveContainer" containerID="e23cbb47dc71cf96f87b4cdac8e8cf6c7805376abdffb3b3f0d481b050bb7107"
Jul 17 22:17:05.689363 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:05.689266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x_odh-ai-gateway-infra(8c778bdf-80c8-4bfc-ae54-cd894be6205d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d"
Jul 17 22:17:06.696769 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:06.696742    2563 scope.go:117] "RemoveContainer" containerID="e23cbb47dc71cf96f87b4cdac8e8cf6c7805376abdffb3b3f0d481b050bb7107"
Jul 17 22:17:06.697154 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:06.696922    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x_odh-ai-gateway-infra(8c778bdf-80c8-4bfc-ae54-cd894be6205d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d"
Jul 17 22:17:11.596665 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:11.596625    2563 scope.go:117] "RemoveContainer" containerID="a6529287c0e221a98bb8dff71e953ee75ca1fd3495693434642fa36355ab7cd8"
Jul 17 22:17:12.729462 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:12.729427    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c842ed7-43de-44f2-a748-f633affba9a2" containerID="430a529bdbe3cba0b64f6c5775d0adbd63eb1f9a6be408db92253bd9f18708a2" exitCode=6
Jul 17 22:17:12.729937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:12.729499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" event={"ID":"6c842ed7-43de-44f2-a748-f633affba9a2","Type":"ContainerDied","Data":"430a529bdbe3cba0b64f6c5775d0adbd63eb1f9a6be408db92253bd9f18708a2"}
Jul 17 22:17:12.729937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:12.729560    2563 scope.go:117] "RemoveContainer" containerID="a6529287c0e221a98bb8dff71e953ee75ca1fd3495693434642fa36355ab7cd8"
Jul 17 22:17:12.729937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:12.729914    2563 scope.go:117] "RemoveContainer" containerID="430a529bdbe3cba0b64f6c5775d0adbd63eb1f9a6be408db92253bd9f18708a2"
Jul 17 22:17:12.730174 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:12.730151    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh_odh-ai-gateway-infra(6c842ed7-43de-44f2-a748-f633affba9a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" podUID="6c842ed7-43de-44f2-a748-f633affba9a2"
Jul 17 22:17:13.763627 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:13.762319    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"]
Jul 17 22:17:13.912922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:13.912889    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:17:14.037802 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.037706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c842ed7-43de-44f2-a748-f633affba9a2-maas-api-service-ca\") pod \"6c842ed7-43de-44f2-a748-f633affba9a2\" (UID: \"6c842ed7-43de-44f2-a748-f633affba9a2\") "
Jul 17 22:17:14.038061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.038038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c842ed7-43de-44f2-a748-f633affba9a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c842ed7-43de-44f2-a748-f633affba9a2" (UID: "6c842ed7-43de-44f2-a748-f633affba9a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:14.138897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.138847    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c842ed7-43de-44f2-a748-f633affba9a2-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:17:14.596790 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.596757    2563 scope.go:117] "RemoveContainer" containerID="154df00a6b6b5ef86b2ab3750d59ca904efe7a37019c4ffd42b349b40a161d02"
Jul 17 22:17:14.742926 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.742889    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"
Jul 17 22:17:14.742926 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.742908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh" event={"ID":"6c842ed7-43de-44f2-a748-f633affba9a2","Type":"ContainerDied","Data":"1b7e6024d07e65392244ad7b53f755ccc49779d484f0a5e859ea2ea9b48c9c5c"}
Jul 17 22:17:14.743170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.742971    2563 scope.go:117] "RemoveContainer" containerID="430a529bdbe3cba0b64f6c5775d0adbd63eb1f9a6be408db92253bd9f18708a2"
Jul 17 22:17:14.765196 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.765166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"]
Jul 17 22:17:14.774770 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:14.774746    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-bn6fh"]
Jul 17 22:17:15.749452 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:15.749422    2563 generic.go:356] "Generic (PLEG): container finished" podID="f929c389-f747-4432-a998-e3197fb143ca" containerID="aaa257ee4eafeec3da7598647c253e298d20f5147f73fb394794eedcdf3f5384" exitCode=6
Jul 17 22:17:15.749657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:15.749497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" event={"ID":"f929c389-f747-4432-a998-e3197fb143ca","Type":"ContainerDied","Data":"aaa257ee4eafeec3da7598647c253e298d20f5147f73fb394794eedcdf3f5384"}
Jul 17 22:17:15.749657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:15.749565    2563 scope.go:117] "RemoveContainer" containerID="154df00a6b6b5ef86b2ab3750d59ca904efe7a37019c4ffd42b349b40a161d02"
Jul 17 22:17:15.749915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:15.749884    2563 scope.go:117] "RemoveContainer" containerID="aaa257ee4eafeec3da7598647c253e298d20f5147f73fb394794eedcdf3f5384"
Jul 17 22:17:15.750113 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:15.750086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl_odh-ai-gateway-infra(f929c389-f747-4432-a998-e3197fb143ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" podUID="f929c389-f747-4432-a998-e3197fb143ca"
Jul 17 22:17:16.603019 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:16.602987    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" path="/var/lib/kubelet/pods/6c842ed7-43de-44f2-a748-f633affba9a2/volumes"
Jul 17 22:17:16.775010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:16.774980    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"]
Jul 17 22:17:16.786196 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:16.786172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:17:16.797943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:16.797909    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:17:16.925373 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:16.925347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:17.061940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.061893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f929c389-f747-4432-a998-e3197fb143ca-maas-api-service-ca\") pod \"f929c389-f747-4432-a998-e3197fb143ca\" (UID: \"f929c389-f747-4432-a998-e3197fb143ca\") "
Jul 17 22:17:17.062276 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.062250    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f929c389-f747-4432-a998-e3197fb143ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f929c389-f747-4432-a998-e3197fb143ca" (UID: "f929c389-f747-4432-a998-e3197fb143ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:17.163198 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.163174    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f929c389-f747-4432-a998-e3197fb143ca-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:17:17.763659 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.763634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"
Jul 17 22:17:17.764064 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.763633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl" event={"ID":"f929c389-f747-4432-a998-e3197fb143ca","Type":"ContainerDied","Data":"59977144ea03d66ad11b0998ac90a4f30e35d835788a767b6989588cc83850f5"}
Jul 17 22:17:17.764064 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.763772    2563 scope.go:117] "RemoveContainer" containerID="aaa257ee4eafeec3da7598647c253e298d20f5147f73fb394794eedcdf3f5384"
Jul 17 22:17:17.787362 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.787338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"]
Jul 17 22:17:17.793023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:17.793001    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-pt7tl"]
Jul 17 22:17:18.088805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.088726    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"]
Jul 17 22:17:18.089097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089101    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089108    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089114    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089124    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089131    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089140    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089145    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089158    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089163    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089170    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089175    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089227    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089234    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089241    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.089348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.089253    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f929c389-f747-4432-a998-e3197fb143ca" containerName="revoke-keys"
Jul 17 22:17:18.093820 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.093802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:18.099359 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.099330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"]
Jul 17 22:17:18.273458 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.273425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be1f831-0da1-4a9e-a12f-afa914ac7210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw\" (UID: \"5be1f831-0da1-4a9e-a12f-afa914ac7210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:18.374897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.374818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be1f831-0da1-4a9e-a12f-afa914ac7210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw\" (UID: \"5be1f831-0da1-4a9e-a12f-afa914ac7210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:18.375524 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.375504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be1f831-0da1-4a9e-a12f-afa914ac7210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw\" (UID: \"5be1f831-0da1-4a9e-a12f-afa914ac7210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:18.405893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.405860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:18.547452 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.547428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"]
Jul 17 22:17:18.549066 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:18.549034    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be1f831_0da1_4a9e_a12f_afa914ac7210.slice/crio-45cf97704a2aeed48dd97cb800f5ff97c9e3cff0a9c99de749a4cc896859911c WatchSource:0}: Error finding container 45cf97704a2aeed48dd97cb800f5ff97c9e3cff0a9c99de749a4cc896859911c: Status 404 returned error can't find the container with id 45cf97704a2aeed48dd97cb800f5ff97c9e3cff0a9c99de749a4cc896859911c
Jul 17 22:17:18.602961 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.602934    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f929c389-f747-4432-a998-e3197fb143ca" path="/var/lib/kubelet/pods/f929c389-f747-4432-a998-e3197fb143ca/volumes"
Jul 17 22:17:18.771036 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:18.771006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" event={"ID":"5be1f831-0da1-4a9e-a12f-afa914ac7210","Type":"ContainerStarted","Data":"45cf97704a2aeed48dd97cb800f5ff97c9e3cff0a9c99de749a4cc896859911c"}
Jul 17 22:17:19.777041 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:19.777007    2563 generic.go:356] "Generic (PLEG): container finished" podID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerID="a57c013546ac00d92872b2421ffcb56ae6928a009979bd71493ab1bfef729085" exitCode=6
Jul 17 22:17:19.777453 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:19.777098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" event={"ID":"5be1f831-0da1-4a9e-a12f-afa914ac7210","Type":"ContainerDied","Data":"a57c013546ac00d92872b2421ffcb56ae6928a009979bd71493ab1bfef729085"}
Jul 17 22:17:19.777453 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:19.777260    2563 scope.go:117] "RemoveContainer" containerID="a57c013546ac00d92872b2421ffcb56ae6928a009979bd71493ab1bfef729085"
Jul 17 22:17:20.603671 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.603646    2563 scope.go:117] "RemoveContainer" containerID="e23cbb47dc71cf96f87b4cdac8e8cf6c7805376abdffb3b3f0d481b050bb7107"
Jul 17 22:17:20.783861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.783828    2563 generic.go:356] "Generic (PLEG): container finished" podID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerID="9819865d4e0187a37a4ea30bc5f0c5044dcc75f38c294fa83dd342cc97cf0ea4" exitCode=6
Jul 17 22:17:20.784270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.783912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" event={"ID":"5be1f831-0da1-4a9e-a12f-afa914ac7210","Type":"ContainerDied","Data":"9819865d4e0187a37a4ea30bc5f0c5044dcc75f38c294fa83dd342cc97cf0ea4"}
Jul 17 22:17:20.784270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.783973    2563 scope.go:117] "RemoveContainer" containerID="a57c013546ac00d92872b2421ffcb56ae6928a009979bd71493ab1bfef729085"
Jul 17 22:17:20.784270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.784131    2563 scope.go:117] "RemoveContainer" containerID="9819865d4e0187a37a4ea30bc5f0c5044dcc75f38c294fa83dd342cc97cf0ea4"
Jul 17 22:17:20.784448 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:20.784364    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw_odh-ai-gateway-infra(5be1f831-0da1-4a9e-a12f-afa914ac7210)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210"
Jul 17 22:17:20.785990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.785970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" event={"ID":"8c778bdf-80c8-4bfc-ae54-cd894be6205d","Type":"ContainerStarted","Data":"3de1937facafcd9d7779714082a178b38c74435d5ae430ebf1cc2feaa6697b29"}
Jul 17 22:17:20.813890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:20.813843    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" podStartSLOduration=17.813828088 podStartE2EDuration="17.813828088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:20.811825372 +0000 UTC m=+3004.724065661" watchObservedRunningTime="2026-07-17 22:17:20.813828088 +0000 UTC m=+3004.726068376"
Jul 17 22:17:21.792526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:21.792489    2563 scope.go:117] "RemoveContainer" containerID="9819865d4e0187a37a4ea30bc5f0c5044dcc75f38c294fa83dd342cc97cf0ea4"
Jul 17 22:17:21.792955 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:21.792727    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw_odh-ai-gateway-infra(5be1f831-0da1-4a9e-a12f-afa914ac7210)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210"
Jul 17 22:17:21.794079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:21.794054    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerID="3de1937facafcd9d7779714082a178b38c74435d5ae430ebf1cc2feaa6697b29" exitCode=6
Jul 17 22:17:21.794190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:21.794129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" event={"ID":"8c778bdf-80c8-4bfc-ae54-cd894be6205d","Type":"ContainerDied","Data":"3de1937facafcd9d7779714082a178b38c74435d5ae430ebf1cc2feaa6697b29"}
Jul 17 22:17:21.794247 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:21.794212    2563 scope.go:117] "RemoveContainer" containerID="e23cbb47dc71cf96f87b4cdac8e8cf6c7805376abdffb3b3f0d481b050bb7107"
Jul 17 22:17:21.794467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:21.794454    2563 scope.go:117] "RemoveContainer" containerID="3de1937facafcd9d7779714082a178b38c74435d5ae430ebf1cc2feaa6697b29"
Jul 17 22:17:21.794676 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:21.794656    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x_odh-ai-gateway-infra(8c778bdf-80c8-4bfc-ae54-cd894be6205d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d"
Jul 17 22:17:21.820313 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:21.820283    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"]
Jul 17 22:17:22.940494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:22.940466    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:23.115324 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.115231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c778bdf-80c8-4bfc-ae54-cd894be6205d-maas-api-service-ca\") pod \"8c778bdf-80c8-4bfc-ae54-cd894be6205d\" (UID: \"8c778bdf-80c8-4bfc-ae54-cd894be6205d\") "
Jul 17 22:17:23.115614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.115572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c778bdf-80c8-4bfc-ae54-cd894be6205d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c778bdf-80c8-4bfc-ae54-cd894be6205d" (UID: "8c778bdf-80c8-4bfc-ae54-cd894be6205d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:23.216089 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.216058    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c778bdf-80c8-4bfc-ae54-cd894be6205d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:17:23.808850 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.808817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"
Jul 17 22:17:23.808850 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.808844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x" event={"ID":"8c778bdf-80c8-4bfc-ae54-cd894be6205d","Type":"ContainerDied","Data":"f197196feeb9a3f7ff6e00815769cca85495b2b9bc2d8847c492d0837c78c8ad"}
Jul 17 22:17:23.809079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.808905    2563 scope.go:117] "RemoveContainer" containerID="3de1937facafcd9d7779714082a178b38c74435d5ae430ebf1cc2feaa6697b29"
Jul 17 22:17:23.829717 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.829689    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"]
Jul 17 22:17:23.833366 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:23.833344    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-jrg5x"]
Jul 17 22:17:24.603043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:24.603010    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" path="/var/lib/kubelet/pods/8c778bdf-80c8-4bfc-ae54-cd894be6205d/volumes"
Jul 17 22:17:26.504356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.504320    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"]
Jul 17 22:17:26.504934 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.504913    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.504936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.504957    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.504966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.504990    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.505000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505262 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.505094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505262 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.505110    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:26.505262 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.505120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c842ed7-43de-44f2-a748-f633affba9a2" containerName="revoke-keys"
Jul 17 22:17:26.508316 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.508295    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.510765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.510734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"kube-root-ca.crt\""
Jul 17 22:17:26.511704 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.511682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"auth-test-model-8e6a3a-kserve-self-signed-certs\""
Jul 17 22:17:26.511801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.511685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"default-dockercfg-g799l\""
Jul 17 22:17:26.511801 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.511736    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"openshift-service-ca.crt\""
Jul 17 22:17:26.517569 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.517547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"]
Jul 17 22:17:26.644956 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.644925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tmp-dir\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.645161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.645012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tls-certs\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.645161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.645052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-model-cache\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.645161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.645128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27z2\" (UniqueName: \"kubernetes.io/projected/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-kube-api-access-l27z2\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.645324 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.645171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-home\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.645324 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.645201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-dshm\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.745915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.745874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27z2\" (UniqueName: \"kubernetes.io/projected/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-kube-api-access-l27z2\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.745915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.745921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-home\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.745949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-dshm\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.745999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tmp-dir\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.746062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tls-certs\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.746111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-model-cache\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746370 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.746309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-home\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746618 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.746574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-model-cache\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.746716 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.746661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tmp-dir\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.748529 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.748507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-dshm\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.748753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.748736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tls-certs\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.754131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.754110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27z2\" (UniqueName: \"kubernetes.io/projected/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-kube-api-access-l27z2\") pod \"auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") " pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.820313 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.820206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:26.959490 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:26.959461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"]
Jul 17 22:17:26.961846 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:26.961811    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae2f15f7_ee18_4863_a8fb_e8430f6f9763.slice/crio-2e581dde39aac6946bb9efa6726337741b4f9935577c07512c472a1e54e904db WatchSource:0}: Error finding container 2e581dde39aac6946bb9efa6726337741b4f9935577c07512c472a1e54e904db: Status 404 returned error can't find the container with id 2e581dde39aac6946bb9efa6726337741b4f9935577c07512c472a1e54e904db
Jul 17 22:17:27.832270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:27.832224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk" event={"ID":"ae2f15f7-ee18-4863-a8fb-e8430f6f9763","Type":"ContainerStarted","Data":"e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887"}
Jul 17 22:17:27.832270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:27.832279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk" event={"ID":"ae2f15f7-ee18-4863-a8fb-e8430f6f9763","Type":"ContainerStarted","Data":"2e581dde39aac6946bb9efa6726337741b4f9935577c07512c472a1e54e904db"}
Jul 17 22:17:27.851299 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:27.851240    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk" podStartSLOduration=1.851219043 podStartE2EDuration="1.851219043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:27.848898602 +0000 UTC m=+3011.761138892" watchObservedRunningTime="2026-07-17 22:17:27.851219043 +0000 UTC m=+3011.763459333"
Jul 17 22:17:36.821037 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:36.821001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:36.821515 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:36.821054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:36.833898 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:36.833868    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:36.902000 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:36.901974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:17:37.596831 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:37.596798    2563 scope.go:117] "RemoveContainer" containerID="9819865d4e0187a37a4ea30bc5f0c5044dcc75f38c294fa83dd342cc97cf0ea4"
Jul 17 22:17:37.897676 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:37.897640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" event={"ID":"5be1f831-0da1-4a9e-a12f-afa914ac7210","Type":"ContainerStarted","Data":"15f1530a04f808a5e467c8177a2f966cac972235515185f0a83e94e29bb3f972"}
Jul 17 22:17:37.913901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:37.913849    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" podStartSLOduration=19.91383561 podStartE2EDuration="19.91383561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:37.911018031 +0000 UTC m=+3021.823258319" watchObservedRunningTime="2026-07-17 22:17:37.91383561 +0000 UTC m=+3021.826075899"
Jul 17 22:17:38.904346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:38.904313    2563 generic.go:356] "Generic (PLEG): container finished" podID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerID="15f1530a04f808a5e467c8177a2f966cac972235515185f0a83e94e29bb3f972" exitCode=6
Jul 17 22:17:38.904845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:38.904386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" event={"ID":"5be1f831-0da1-4a9e-a12f-afa914ac7210","Type":"ContainerDied","Data":"15f1530a04f808a5e467c8177a2f966cac972235515185f0a83e94e29bb3f972"}
Jul 17 22:17:38.904845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:38.904444    2563 scope.go:117] "RemoveContainer" containerID="9819865d4e0187a37a4ea30bc5f0c5044dcc75f38c294fa83dd342cc97cf0ea4"
Jul 17 22:17:38.904845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:38.904722    2563 scope.go:117] "RemoveContainer" containerID="15f1530a04f808a5e467c8177a2f966cac972235515185f0a83e94e29bb3f972"
Jul 17 22:17:38.905060 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:38.904968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw_odh-ai-gateway-infra(5be1f831-0da1-4a9e-a12f-afa914ac7210)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210"
Jul 17 22:17:38.927486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:38.927456    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"]
Jul 17 22:17:40.055752 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.055728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:40.152011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.151975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be1f831-0da1-4a9e-a12f-afa914ac7210-maas-api-service-ca\") pod \"5be1f831-0da1-4a9e-a12f-afa914ac7210\" (UID: \"5be1f831-0da1-4a9e-a12f-afa914ac7210\") "
Jul 17 22:17:40.152380 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.152358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be1f831-0da1-4a9e-a12f-afa914ac7210-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5be1f831-0da1-4a9e-a12f-afa914ac7210" (UID: "5be1f831-0da1-4a9e-a12f-afa914ac7210"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:40.253307 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.253226    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be1f831-0da1-4a9e-a12f-afa914ac7210-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:17:40.918056 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.918028    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"
Jul 17 22:17:40.918240 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.918030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw" event={"ID":"5be1f831-0da1-4a9e-a12f-afa914ac7210","Type":"ContainerDied","Data":"45cf97704a2aeed48dd97cb800f5ff97c9e3cff0a9c99de749a4cc896859911c"}
Jul 17 22:17:40.918240 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.918164    2563 scope.go:117] "RemoveContainer" containerID="15f1530a04f808a5e467c8177a2f966cac972235515185f0a83e94e29bb3f972"
Jul 17 22:17:40.933486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.933462    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"]
Jul 17 22:17:40.938515 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:40.938495    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-872dw"]
Jul 17 22:17:42.602189 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.602158    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" path="/var/lib/kubelet/pods/5be1f831-0da1-4a9e-a12f-afa914ac7210/volumes"
Jul 17 22:17:42.846582 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.846549    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"]
Jul 17 22:17:42.847007 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.846989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847009    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847022    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847030    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847061    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847096 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847072    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847169    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c778bdf-80c8-4bfc-ae54-cd894be6205d" containerName="revoke-keys"
Jul 17 22:17:42.847351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847183    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847193    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.847351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.847202    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be1f831-0da1-4a9e-a12f-afa914ac7210" containerName="revoke-keys"
Jul 17 22:17:42.851578 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.851555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.855811 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.855732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"default-dockercfg-99xqt\""
Jul 17 22:17:42.856525 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.856500    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"openshift-service-ca.crt\""
Jul 17 22:17:42.856653 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.856526    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"kube-root-ca.crt\""
Jul 17 22:17:42.856653 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.856500    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"auth-test-model-5401e5-kserve-self-signed-certs\""
Jul 17 22:17:42.862141 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.862115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"]
Jul 17 22:17:42.873850 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.873817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-dshm\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.873969 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.873877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd96\" (UniqueName: \"kubernetes.io/projected/a943a051-83e5-485f-ac24-1a5c0b8b463e-kube-api-access-5kd96\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.873969 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.873909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-home\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.874097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.873968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a943a051-83e5-485f-ac24-1a5c0b8b463e-tls-certs\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.874097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.874064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-tmp-dir\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.874353 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.874104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-model-cache\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975521 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-model-cache\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975521 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-dshm\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975758 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd96\" (UniqueName: \"kubernetes.io/projected/a943a051-83e5-485f-ac24-1a5c0b8b463e-kube-api-access-5kd96\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975758 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-home\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975860 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a943a051-83e5-485f-ac24-1a5c0b8b463e-tls-certs\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975860 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-tmp-dir\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.975994 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.975971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-model-cache\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.976086 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.976063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-home\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.976147 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.976109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-tmp-dir\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.978078 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.978050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-dshm\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.978441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.978421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a943a051-83e5-485f-ac24-1a5c0b8b463e-tls-certs\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:42.988373 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:42.988353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd96\" (UniqueName: \"kubernetes.io/projected/a943a051-83e5-485f-ac24-1a5c0b8b463e-kube-api-access-5kd96\") pod \"auth-test-model-5401e5-kserve-785f76d9d7-qgrp5\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") " pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:43.161677 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:43.161644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:43.304259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:43.304227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"]
Jul 17 22:17:43.305940 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:43.305916    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda943a051_83e5_485f_ac24_1a5c0b8b463e.slice/crio-2d04e5ce139a48bc214990455d21e758c2ed860ab15f1fe5242c5357201db6e5 WatchSource:0}: Error finding container 2d04e5ce139a48bc214990455d21e758c2ed860ab15f1fe5242c5357201db6e5: Status 404 returned error can't find the container with id 2d04e5ce139a48bc214990455d21e758c2ed860ab15f1fe5242c5357201db6e5
Jul 17 22:17:43.935062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:43.935027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5" event={"ID":"a943a051-83e5-485f-ac24-1a5c0b8b463e","Type":"ContainerStarted","Data":"f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64"}
Jul 17 22:17:43.935062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:43.935069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5" event={"ID":"a943a051-83e5-485f-ac24-1a5c0b8b463e","Type":"ContainerStarted","Data":"2d04e5ce139a48bc214990455d21e758c2ed860ab15f1fe5242c5357201db6e5"}
Jul 17 22:17:43.955417 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:43.955368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5" podStartSLOduration=1.955352908 podStartE2EDuration="1.955352908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:43.952984032 +0000 UTC m=+3027.865224344" watchObservedRunningTime="2026-07-17 22:17:43.955352908 +0000 UTC m=+3027.867593196"
Jul 17 22:17:46.005627 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.005556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"]
Jul 17 22:17:46.009351 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.009328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:17:46.011895 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.011872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:17:46.017069 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.016996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"]
Jul 17 22:17:46.101788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.101753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e34015-cc0a-4c44-bc9e-3f378b8195b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht\" (UID: \"72e34015-cc0a-4c44-bc9e-3f378b8195b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:17:46.202694 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.202666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e34015-cc0a-4c44-bc9e-3f378b8195b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht\" (UID: \"72e34015-cc0a-4c44-bc9e-3f378b8195b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:17:46.203343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.203323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e34015-cc0a-4c44-bc9e-3f378b8195b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht\" (UID: \"72e34015-cc0a-4c44-bc9e-3f378b8195b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:17:46.322431 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.322348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:17:46.667487 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.667461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"]
Jul 17 22:17:46.668828 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:46.668805    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e34015_cc0a_4c44_bc9e_3f378b8195b9.slice/crio-4831ecfcf3338748632a88a5ecb6484460d7fbf1f8c198a6a08e61d8b4fa03ef WatchSource:0}: Error finding container 4831ecfcf3338748632a88a5ecb6484460d7fbf1f8c198a6a08e61d8b4fa03ef: Status 404 returned error can't find the container with id 4831ecfcf3338748632a88a5ecb6484460d7fbf1f8c198a6a08e61d8b4fa03ef
Jul 17 22:17:46.950797 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:46.950705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" event={"ID":"72e34015-cc0a-4c44-bc9e-3f378b8195b9","Type":"ContainerStarted","Data":"4831ecfcf3338748632a88a5ecb6484460d7fbf1f8c198a6a08e61d8b4fa03ef"}
Jul 17 22:17:47.958031 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:47.957988    2563 generic.go:356] "Generic (PLEG): container finished" podID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerID="1c90dfb42a14d9cd6c4fc4bc86d954c98d26cfc5aa87563492a7b2c4a1d3ec0d" exitCode=6
Jul 17 22:17:47.958446 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:47.958041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" event={"ID":"72e34015-cc0a-4c44-bc9e-3f378b8195b9","Type":"ContainerDied","Data":"1c90dfb42a14d9cd6c4fc4bc86d954c98d26cfc5aa87563492a7b2c4a1d3ec0d"}
Jul 17 22:17:47.958446 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:47.958280    2563 scope.go:117] "RemoveContainer" containerID="1c90dfb42a14d9cd6c4fc4bc86d954c98d26cfc5aa87563492a7b2c4a1d3ec0d"
Jul 17 22:17:48.965559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:48.965530    2563 generic.go:356] "Generic (PLEG): container finished" podID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerID="8feeea06ef85b866240e4e6121f62afcb88aac0fab8c21cafc86771af8b455af" exitCode=6
Jul 17 22:17:48.966001 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:48.965630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" event={"ID":"72e34015-cc0a-4c44-bc9e-3f378b8195b9","Type":"ContainerDied","Data":"8feeea06ef85b866240e4e6121f62afcb88aac0fab8c21cafc86771af8b455af"}
Jul 17 22:17:48.966001 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:48.965699    2563 scope.go:117] "RemoveContainer" containerID="1c90dfb42a14d9cd6c4fc4bc86d954c98d26cfc5aa87563492a7b2c4a1d3ec0d"
Jul 17 22:17:48.966001 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:48.965911    2563 scope.go:117] "RemoveContainer" containerID="8feeea06ef85b866240e4e6121f62afcb88aac0fab8c21cafc86771af8b455af"
Jul 17 22:17:48.966176 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:48.966155    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht_odh-ai-gateway-infra(72e34015-cc0a-4c44-bc9e-3f378b8195b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9"
Jul 17 22:17:49.972424 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:49.972396    2563 scope.go:117] "RemoveContainer" containerID="8feeea06ef85b866240e4e6121f62afcb88aac0fab8c21cafc86771af8b455af"
Jul 17 22:17:49.972812 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:49.972617    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht_odh-ai-gateway-infra(72e34015-cc0a-4c44-bc9e-3f378b8195b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9"
Jul 17 22:17:50.324674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.324576    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"]
Jul 17 22:17:50.329298 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.329282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:17:50.334051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.334028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"]
Jul 17 22:17:50.446480 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.446434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748ce9e9-f6a5-4972-944e-d18f8ea7224d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk\" (UID: \"748ce9e9-f6a5-4972-944e-d18f8ea7224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:17:50.547420 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.547384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748ce9e9-f6a5-4972-944e-d18f8ea7224d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk\" (UID: \"748ce9e9-f6a5-4972-944e-d18f8ea7224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:17:50.548039 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.548020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748ce9e9-f6a5-4972-944e-d18f8ea7224d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk\" (UID: \"748ce9e9-f6a5-4972-944e-d18f8ea7224d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:17:50.641480 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.641404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:17:50.778617 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.778572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"]
Jul 17 22:17:50.779946 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:17:50.779914    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748ce9e9_f6a5_4972_944e_d18f8ea7224d.slice/crio-9dcfd61a103ff36f6bed9565e191777f35d9615c4b876de91bf387b23aee4c1c WatchSource:0}: Error finding container 9dcfd61a103ff36f6bed9565e191777f35d9615c4b876de91bf387b23aee4c1c: Status 404 returned error can't find the container with id 9dcfd61a103ff36f6bed9565e191777f35d9615c4b876de91bf387b23aee4c1c
Jul 17 22:17:50.979906 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:50.979877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" event={"ID":"748ce9e9-f6a5-4972-944e-d18f8ea7224d","Type":"ContainerStarted","Data":"9dcfd61a103ff36f6bed9565e191777f35d9615c4b876de91bf387b23aee4c1c"}
Jul 17 22:17:51.987007 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:51.986973    2563 generic.go:356] "Generic (PLEG): container finished" podID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerID="6e5ae1e1adcf7a90a08c2f7e1f559f8a10a65839d2d5c6b9cbced637c32ee36e" exitCode=6
Jul 17 22:17:51.987428 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:51.987062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" event={"ID":"748ce9e9-f6a5-4972-944e-d18f8ea7224d","Type":"ContainerDied","Data":"6e5ae1e1adcf7a90a08c2f7e1f559f8a10a65839d2d5c6b9cbced637c32ee36e"}
Jul 17 22:17:51.987428 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:51.987230    2563 scope.go:117] "RemoveContainer" containerID="6e5ae1e1adcf7a90a08c2f7e1f559f8a10a65839d2d5c6b9cbced637c32ee36e"
Jul 17 22:17:52.994665 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:52.994628    2563 generic.go:356] "Generic (PLEG): container finished" podID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerID="f28b20f8f1ab2f085b9e6992e213236a15d013294fcb106050cff4cd767727b0" exitCode=6
Jul 17 22:17:52.995107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:52.994714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" event={"ID":"748ce9e9-f6a5-4972-944e-d18f8ea7224d","Type":"ContainerDied","Data":"f28b20f8f1ab2f085b9e6992e213236a15d013294fcb106050cff4cd767727b0"}
Jul 17 22:17:52.995107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:52.994775    2563 scope.go:117] "RemoveContainer" containerID="6e5ae1e1adcf7a90a08c2f7e1f559f8a10a65839d2d5c6b9cbced637c32ee36e"
Jul 17 22:17:52.995107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:52.994952    2563 scope.go:117] "RemoveContainer" containerID="f28b20f8f1ab2f085b9e6992e213236a15d013294fcb106050cff4cd767727b0"
Jul 17 22:17:52.995242 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:52.995174    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk_odh-ai-gateway-infra(748ce9e9-f6a5-4972-944e-d18f8ea7224d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d"
Jul 17 22:17:53.162633 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:53.162576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:53.162834 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:53.162648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:53.175310 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:53.175287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:17:54.001492 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:54.001459    2563 scope.go:117] "RemoveContainer" containerID="f28b20f8f1ab2f085b9e6992e213236a15d013294fcb106050cff4cd767727b0"
Jul 17 22:17:54.001920 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:17:54.001693    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk_odh-ai-gateway-infra(748ce9e9-f6a5-4972-944e-d18f8ea7224d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d"
Jul 17 22:17:54.012711 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:17:54.012687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:18:00.597113 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:00.597075    2563 scope.go:117] "RemoveContainer" containerID="8feeea06ef85b866240e4e6121f62afcb88aac0fab8c21cafc86771af8b455af"
Jul 17 22:18:01.040137 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:01.040103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" event={"ID":"72e34015-cc0a-4c44-bc9e-3f378b8195b9","Type":"ContainerStarted","Data":"962a910afc3d0d1e01947a25fffb295cedab58ae3e90590d7362ccaff016a33f"}
Jul 17 22:18:01.055479 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:01.055434    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" podStartSLOduration=16.055420704 podStartE2EDuration="16.055420704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:01.053896314 +0000 UTC m=+3044.966136618" watchObservedRunningTime="2026-07-17 22:18:01.055420704 +0000 UTC m=+3044.967660994"
Jul 17 22:18:02.047113 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:02.047070    2563 generic.go:356] "Generic (PLEG): container finished" podID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerID="962a910afc3d0d1e01947a25fffb295cedab58ae3e90590d7362ccaff016a33f" exitCode=6
Jul 17 22:18:02.047591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:02.047112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" event={"ID":"72e34015-cc0a-4c44-bc9e-3f378b8195b9","Type":"ContainerDied","Data":"962a910afc3d0d1e01947a25fffb295cedab58ae3e90590d7362ccaff016a33f"}
Jul 17 22:18:02.047591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:02.047181    2563 scope.go:117] "RemoveContainer" containerID="8feeea06ef85b866240e4e6121f62afcb88aac0fab8c21cafc86771af8b455af"
Jul 17 22:18:02.047591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:02.047439    2563 scope.go:117] "RemoveContainer" containerID="962a910afc3d0d1e01947a25fffb295cedab58ae3e90590d7362ccaff016a33f"
Jul 17 22:18:02.047781 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:02.047680    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht_odh-ai-gateway-infra(72e34015-cc0a-4c44-bc9e-3f378b8195b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9"
Jul 17 22:18:02.070149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:02.070121    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"]
Jul 17 22:18:03.191070 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.191044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:18:03.257618 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.257568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e34015-cc0a-4c44-bc9e-3f378b8195b9-maas-api-service-ca\") pod \"72e34015-cc0a-4c44-bc9e-3f378b8195b9\" (UID: \"72e34015-cc0a-4c44-bc9e-3f378b8195b9\") "
Jul 17 22:18:03.257895 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.257873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e34015-cc0a-4c44-bc9e-3f378b8195b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72e34015-cc0a-4c44-bc9e-3f378b8195b9" (UID: "72e34015-cc0a-4c44-bc9e-3f378b8195b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:03.359033 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.358951    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e34015-cc0a-4c44-bc9e-3f378b8195b9-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:18:03.743300 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743257    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"]
Jul 17 22:18:03.743751 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743731    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:03.743845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743753    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:03.743845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743770    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:03.743845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743777    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:03.744017 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:03.744017 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.743915    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:03.747151 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.747130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:03.754960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.754934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"]
Jul 17 22:18:03.862983 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.862946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6e8b5-2cd5-4e78-b50c-152afc3a6251-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws\" (UID: \"cce6e8b5-2cd5-4e78-b50c-152afc3a6251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:03.964190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.964154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6e8b5-2cd5-4e78-b50c-152afc3a6251-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws\" (UID: \"cce6e8b5-2cd5-4e78-b50c-152afc3a6251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:03.964876 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:03.964853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6e8b5-2cd5-4e78-b50c-152afc3a6251-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws\" (UID: \"cce6e8b5-2cd5-4e78-b50c-152afc3a6251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:04.059331 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.059238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:04.060577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.060555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"
Jul 17 22:18:04.060577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.060562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht" event={"ID":"72e34015-cc0a-4c44-bc9e-3f378b8195b9","Type":"ContainerDied","Data":"4831ecfcf3338748632a88a5ecb6484460d7fbf1f8c198a6a08e61d8b4fa03ef"}
Jul 17 22:18:04.060782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.060651    2563 scope.go:117] "RemoveContainer" containerID="962a910afc3d0d1e01947a25fffb295cedab58ae3e90590d7362ccaff016a33f"
Jul 17 22:18:04.102403 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.102370    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"]
Jul 17 22:18:04.109477 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.109444    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-m8vht"]
Jul 17 22:18:04.415246 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.415223    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"]
Jul 17 22:18:04.416762 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:18:04.416735    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce6e8b5_2cd5_4e78_b50c_152afc3a6251.slice/crio-a77720f162d159f818eaf5eeed02211491c09d17b4867f88965b7e2b167d856f WatchSource:0}: Error finding container a77720f162d159f818eaf5eeed02211491c09d17b4867f88965b7e2b167d856f: Status 404 returned error can't find the container with id a77720f162d159f818eaf5eeed02211491c09d17b4867f88965b7e2b167d856f
Jul 17 22:18:04.602568 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:04.602529    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" path="/var/lib/kubelet/pods/72e34015-cc0a-4c44-bc9e-3f378b8195b9/volumes"
Jul 17 22:18:05.067267 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:05.067173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" event={"ID":"cce6e8b5-2cd5-4e78-b50c-152afc3a6251","Type":"ContainerStarted","Data":"1e7a2469718db95810477bd349a3aaad0994b64dac513d37b9f02f953ce21254"}
Jul 17 22:18:05.067267 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:05.067262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" event={"ID":"cce6e8b5-2cd5-4e78-b50c-152afc3a6251","Type":"ContainerStarted","Data":"a77720f162d159f818eaf5eeed02211491c09d17b4867f88965b7e2b167d856f"}
Jul 17 22:18:05.083152 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:05.083106    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" podStartSLOduration=2.083091921 podStartE2EDuration="2.083091921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:05.081069563 +0000 UTC m=+3048.993309845" watchObservedRunningTime="2026-07-17 22:18:05.083091921 +0000 UTC m=+3048.995332239"
Jul 17 22:18:06.075429 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:06.075397    2563 generic.go:356] "Generic (PLEG): container finished" podID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerID="1e7a2469718db95810477bd349a3aaad0994b64dac513d37b9f02f953ce21254" exitCode=6
Jul 17 22:18:06.075851 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:06.075457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" event={"ID":"cce6e8b5-2cd5-4e78-b50c-152afc3a6251","Type":"ContainerDied","Data":"1e7a2469718db95810477bd349a3aaad0994b64dac513d37b9f02f953ce21254"}
Jul 17 22:18:06.075851 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:06.075672    2563 scope.go:117] "RemoveContainer" containerID="1e7a2469718db95810477bd349a3aaad0994b64dac513d37b9f02f953ce21254"
Jul 17 22:18:06.599591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:06.599556    2563 scope.go:117] "RemoveContainer" containerID="f28b20f8f1ab2f085b9e6992e213236a15d013294fcb106050cff4cd767727b0"
Jul 17 22:18:07.082312 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:07.082275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" event={"ID":"748ce9e9-f6a5-4972-944e-d18f8ea7224d","Type":"ContainerStarted","Data":"847fa2f50058780168dd4c75f8f42b95c198a286c86bfec16bdaa60a21181e6b"}
Jul 17 22:18:07.084120 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:07.084094    2563 generic.go:356] "Generic (PLEG): container finished" podID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerID="3616f61d8512de93c16b2d0baf822e51bd1798ca5a3a6c556f46e209c9a92352" exitCode=6
Jul 17 22:18:07.084239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:07.084130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" event={"ID":"cce6e8b5-2cd5-4e78-b50c-152afc3a6251","Type":"ContainerDied","Data":"3616f61d8512de93c16b2d0baf822e51bd1798ca5a3a6c556f46e209c9a92352"}
Jul 17 22:18:07.084239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:07.084180    2563 scope.go:117] "RemoveContainer" containerID="1e7a2469718db95810477bd349a3aaad0994b64dac513d37b9f02f953ce21254"
Jul 17 22:18:07.084415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:07.084397    2563 scope.go:117] "RemoveContainer" containerID="3616f61d8512de93c16b2d0baf822e51bd1798ca5a3a6c556f46e209c9a92352"
Jul 17 22:18:07.084641 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:07.084619    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws_odh-ai-gateway-infra(cce6e8b5-2cd5-4e78-b50c-152afc3a6251)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251"
Jul 17 22:18:07.101653 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:07.101615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" podStartSLOduration=17.101585289 podStartE2EDuration="17.101585289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:07.099079637 +0000 UTC m=+3051.011319926" watchObservedRunningTime="2026-07-17 22:18:07.101585289 +0000 UTC m=+3051.013825577"
Jul 17 22:18:08.091142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:08.091064    2563 generic.go:356] "Generic (PLEG): container finished" podID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerID="847fa2f50058780168dd4c75f8f42b95c198a286c86bfec16bdaa60a21181e6b" exitCode=6
Jul 17 22:18:08.091564 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:08.091140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" event={"ID":"748ce9e9-f6a5-4972-944e-d18f8ea7224d","Type":"ContainerDied","Data":"847fa2f50058780168dd4c75f8f42b95c198a286c86bfec16bdaa60a21181e6b"}
Jul 17 22:18:08.091564 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:08.091210    2563 scope.go:117] "RemoveContainer" containerID="f28b20f8f1ab2f085b9e6992e213236a15d013294fcb106050cff4cd767727b0"
Jul 17 22:18:08.091564 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:08.091474    2563 scope.go:117] "RemoveContainer" containerID="847fa2f50058780168dd4c75f8f42b95c198a286c86bfec16bdaa60a21181e6b"
Jul 17 22:18:08.091751 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:08.091717    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk_odh-ai-gateway-infra(748ce9e9-f6a5-4972-944e-d18f8ea7224d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d"
Jul 17 22:18:08.093187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:08.093170    2563 scope.go:117] "RemoveContainer" containerID="3616f61d8512de93c16b2d0baf822e51bd1798ca5a3a6c556f46e209c9a92352"
Jul 17 22:18:08.093343 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:08.093326    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws_odh-ai-gateway-infra(cce6e8b5-2cd5-4e78-b50c-152afc3a6251)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251"
Jul 17 22:18:08.115237 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:08.115212    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"]
Jul 17 22:18:09.245565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:09.245536    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:18:09.412752 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:09.412718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748ce9e9-f6a5-4972-944e-d18f8ea7224d-maas-api-service-ca\") pod \"748ce9e9-f6a5-4972-944e-d18f8ea7224d\" (UID: \"748ce9e9-f6a5-4972-944e-d18f8ea7224d\") "
Jul 17 22:18:09.413097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:09.413069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748ce9e9-f6a5-4972-944e-d18f8ea7224d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "748ce9e9-f6a5-4972-944e-d18f8ea7224d" (UID: "748ce9e9-f6a5-4972-944e-d18f8ea7224d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:09.513453 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:09.513413    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/748ce9e9-f6a5-4972-944e-d18f8ea7224d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:18:10.106337 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:10.106307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"
Jul 17 22:18:10.106541 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:10.106356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk" event={"ID":"748ce9e9-f6a5-4972-944e-d18f8ea7224d","Type":"ContainerDied","Data":"9dcfd61a103ff36f6bed9565e191777f35d9615c4b876de91bf387b23aee4c1c"}
Jul 17 22:18:10.106541 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:10.106419    2563 scope.go:117] "RemoveContainer" containerID="847fa2f50058780168dd4c75f8f42b95c198a286c86bfec16bdaa60a21181e6b"
Jul 17 22:18:10.129868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:10.129837    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"]
Jul 17 22:18:10.135767 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:10.135742    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-jgbrk"]
Jul 17 22:18:10.602678 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:10.602649    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" path="/var/lib/kubelet/pods/748ce9e9-f6a5-4972-944e-d18f8ea7224d/volumes"
Jul 17 22:18:18.142323 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142287    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"]
Jul 17 22:18:18.142763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142689    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:18.142763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:18.142763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142709    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:18.142763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142714    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:18.142763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142728    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:18.142763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142733    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:18.142990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142800    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e34015-cc0a-4c44-bc9e-3f378b8195b9" containerName="revoke-keys"
Jul 17 22:18:18.142990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142808    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:18.142990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.142818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:18.147650 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.147629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:18.152590 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.152561    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"]
Jul 17 22:18:18.287193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.287159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25495407-47dd-4f75-96ec-5d3eb4e72f14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn\" (UID: \"25495407-47dd-4f75-96ec-5d3eb4e72f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:18.388615 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.388563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25495407-47dd-4f75-96ec-5d3eb4e72f14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn\" (UID: \"25495407-47dd-4f75-96ec-5d3eb4e72f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:18.389173 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.389146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25495407-47dd-4f75-96ec-5d3eb4e72f14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn\" (UID: \"25495407-47dd-4f75-96ec-5d3eb4e72f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:18.460420 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.460387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:18.598708 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:18:18.598676    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25495407_47dd_4f75_96ec_5d3eb4e72f14.slice/crio-aed1e7c60c1196ab706aad63fb387119ddc90c3e7b77941e4f5e2e7094896e30 WatchSource:0}: Error finding container aed1e7c60c1196ab706aad63fb387119ddc90c3e7b77941e4f5e2e7094896e30: Status 404 returned error can't find the container with id aed1e7c60c1196ab706aad63fb387119ddc90c3e7b77941e4f5e2e7094896e30
Jul 17 22:18:18.602990 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:18.602969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"]
Jul 17 22:18:19.161938 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:19.161913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" event={"ID":"25495407-47dd-4f75-96ec-5d3eb4e72f14","Type":"ContainerStarted","Data":"2f332ae0904558fba9fe9ac063a215f1be1009d0db2620736db59cf95cc22aa6"}
Jul 17 22:18:19.162304 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:19.161962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" event={"ID":"25495407-47dd-4f75-96ec-5d3eb4e72f14","Type":"ContainerStarted","Data":"aed1e7c60c1196ab706aad63fb387119ddc90c3e7b77941e4f5e2e7094896e30"}
Jul 17 22:18:19.176403 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:19.176352    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" podStartSLOduration=1.176335597 podStartE2EDuration="1.176335597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:19.175303678 +0000 UTC m=+3063.087543966" watchObservedRunningTime="2026-07-17 22:18:19.176335597 +0000 UTC m=+3063.088575885"
Jul 17 22:18:20.168584 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:20.168549    2563 generic.go:356] "Generic (PLEG): container finished" podID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerID="2f332ae0904558fba9fe9ac063a215f1be1009d0db2620736db59cf95cc22aa6" exitCode=6
Jul 17 22:18:20.169008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:20.168630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" event={"ID":"25495407-47dd-4f75-96ec-5d3eb4e72f14","Type":"ContainerDied","Data":"2f332ae0904558fba9fe9ac063a215f1be1009d0db2620736db59cf95cc22aa6"}
Jul 17 22:18:20.169008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:20.168852    2563 scope.go:117] "RemoveContainer" containerID="2f332ae0904558fba9fe9ac063a215f1be1009d0db2620736db59cf95cc22aa6"
Jul 17 22:18:21.175238 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:21.175207    2563 generic.go:356] "Generic (PLEG): container finished" podID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerID="f150050f652f730e0fdd6c3b1833fb7c0e72391631d45aaf100ebc9bf2faee65" exitCode=6
Jul 17 22:18:21.175703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:21.175290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" event={"ID":"25495407-47dd-4f75-96ec-5d3eb4e72f14","Type":"ContainerDied","Data":"f150050f652f730e0fdd6c3b1833fb7c0e72391631d45aaf100ebc9bf2faee65"}
Jul 17 22:18:21.175703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:21.175354    2563 scope.go:117] "RemoveContainer" containerID="2f332ae0904558fba9fe9ac063a215f1be1009d0db2620736db59cf95cc22aa6"
Jul 17 22:18:21.175703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:21.175571    2563 scope.go:117] "RemoveContainer" containerID="f150050f652f730e0fdd6c3b1833fb7c0e72391631d45aaf100ebc9bf2faee65"
Jul 17 22:18:21.175972 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:21.175806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn_odh-ai-gateway-infra(25495407-47dd-4f75-96ec-5d3eb4e72f14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14"
Jul 17 22:18:21.597011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:21.596924    2563 scope.go:117] "RemoveContainer" containerID="3616f61d8512de93c16b2d0baf822e51bd1798ca5a3a6c556f46e209c9a92352"
Jul 17 22:18:22.182119 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:22.182094    2563 scope.go:117] "RemoveContainer" containerID="f150050f652f730e0fdd6c3b1833fb7c0e72391631d45aaf100ebc9bf2faee65"
Jul 17 22:18:22.182487 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:22.182362    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn_odh-ai-gateway-infra(25495407-47dd-4f75-96ec-5d3eb4e72f14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14"
Jul 17 22:18:22.183794 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:22.183770    2563 generic.go:356] "Generic (PLEG): container finished" podID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerID="59a4db250c277a5ce61d86629ec82e63f431317ef3fb6ef35faa22ac578a6980" exitCode=6
Jul 17 22:18:22.183900 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:22.183828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" event={"ID":"cce6e8b5-2cd5-4e78-b50c-152afc3a6251","Type":"ContainerDied","Data":"59a4db250c277a5ce61d86629ec82e63f431317ef3fb6ef35faa22ac578a6980"}
Jul 17 22:18:22.183900 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:22.183885    2563 scope.go:117] "RemoveContainer" containerID="3616f61d8512de93c16b2d0baf822e51bd1798ca5a3a6c556f46e209c9a92352"
Jul 17 22:18:22.184184 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:22.184169    2563 scope.go:117] "RemoveContainer" containerID="59a4db250c277a5ce61d86629ec82e63f431317ef3fb6ef35faa22ac578a6980"
Jul 17 22:18:22.184387 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:22.184366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws_odh-ai-gateway-infra(cce6e8b5-2cd5-4e78-b50c-152afc3a6251)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251"
Jul 17 22:18:23.221017 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:23.220985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"]
Jul 17 22:18:23.367412 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:23.367389    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:23.434511 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:23.434478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6e8b5-2cd5-4e78-b50c-152afc3a6251-maas-api-service-ca\") pod \"cce6e8b5-2cd5-4e78-b50c-152afc3a6251\" (UID: \"cce6e8b5-2cd5-4e78-b50c-152afc3a6251\") "
Jul 17 22:18:23.434886 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:23.434858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce6e8b5-2cd5-4e78-b50c-152afc3a6251-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cce6e8b5-2cd5-4e78-b50c-152afc3a6251" (UID: "cce6e8b5-2cd5-4e78-b50c-152afc3a6251"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:23.535573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:23.535490    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6e8b5-2cd5-4e78-b50c-152afc3a6251-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:18:24.197234 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:24.197191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws" event={"ID":"cce6e8b5-2cd5-4e78-b50c-152afc3a6251","Type":"ContainerDied","Data":"a77720f162d159f818eaf5eeed02211491c09d17b4867f88965b7e2b167d856f"}
Jul 17 22:18:24.197430 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:24.197259    2563 scope.go:117] "RemoveContainer" containerID="59a4db250c277a5ce61d86629ec82e63f431317ef3fb6ef35faa22ac578a6980"
Jul 17 22:18:24.197430 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:24.197216    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"
Jul 17 22:18:24.218845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:24.218820    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"]
Jul 17 22:18:24.222417 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:24.222389    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-cmjws"]
Jul 17 22:18:24.603035 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:24.602958    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" path="/var/lib/kubelet/pods/cce6e8b5-2cd5-4e78-b50c-152afc3a6251/volumes"
Jul 17 22:18:34.597207 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:34.597174    2563 scope.go:117] "RemoveContainer" containerID="f150050f652f730e0fdd6c3b1833fb7c0e72391631d45aaf100ebc9bf2faee65"
Jul 17 22:18:35.256010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:35.255979    2563 generic.go:356] "Generic (PLEG): container finished" podID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerID="500b747c22dfe3fbf1b1d753a9f7916b2641aaad694768cc9d05cc9e23f315a8" exitCode=6
Jul 17 22:18:35.256186 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:35.256057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" event={"ID":"25495407-47dd-4f75-96ec-5d3eb4e72f14","Type":"ContainerDied","Data":"500b747c22dfe3fbf1b1d753a9f7916b2641aaad694768cc9d05cc9e23f315a8"}
Jul 17 22:18:35.256186 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:35.256125    2563 scope.go:117] "RemoveContainer" containerID="f150050f652f730e0fdd6c3b1833fb7c0e72391631d45aaf100ebc9bf2faee65"
Jul 17 22:18:35.256395 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:35.256371    2563 scope.go:117] "RemoveContainer" containerID="500b747c22dfe3fbf1b1d753a9f7916b2641aaad694768cc9d05cc9e23f315a8"
Jul 17 22:18:35.256638 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:35.256615    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn_odh-ai-gateway-infra(25495407-47dd-4f75-96ec-5d3eb4e72f14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14"
Jul 17 22:18:36.041779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.041744    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"]
Jul 17 22:18:36.042149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042104    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042114    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042124    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042129    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042138    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042156    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:36.042343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042162    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:36.042343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042232    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042240    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:36.042343 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.042246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="748ce9e9-f6a5-4972-944e-d18f8ea7224d" containerName="revoke-keys"
Jul 17 22:18:36.046494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.046477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:36.050844 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.050819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"]
Jul 17 22:18:36.139868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.139832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed224be-9547-4b5b-b069-4266c5cf68fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj\" (UID: \"6ed224be-9547-4b5b-b069-4266c5cf68fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:36.240497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.240457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed224be-9547-4b5b-b069-4266c5cf68fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj\" (UID: \"6ed224be-9547-4b5b-b069-4266c5cf68fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:36.241088 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.241070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed224be-9547-4b5b-b069-4266c5cf68fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj\" (UID: \"6ed224be-9547-4b5b-b069-4266c5cf68fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:36.284834 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.284804    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"]
Jul 17 22:18:36.358914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.358889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:36.441545 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.441518    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:36.543937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.543909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25495407-47dd-4f75-96ec-5d3eb4e72f14-maas-api-service-ca\") pod \"25495407-47dd-4f75-96ec-5d3eb4e72f14\" (UID: \"25495407-47dd-4f75-96ec-5d3eb4e72f14\") "
Jul 17 22:18:36.544307 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.544282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25495407-47dd-4f75-96ec-5d3eb4e72f14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25495407-47dd-4f75-96ec-5d3eb4e72f14" (UID: "25495407-47dd-4f75-96ec-5d3eb4e72f14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:36.644577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.644541    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25495407-47dd-4f75-96ec-5d3eb4e72f14-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:18:36.713552 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:36.713528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"]
Jul 17 22:18:36.715478 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:18:36.715453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed224be_9547_4b5b_b069_4266c5cf68fa.slice/crio-3c2f2faa08ee0c5b9bfcbfc8a33768c3d798bdec34654babee9367779ed5fcd4 WatchSource:0}: Error finding container 3c2f2faa08ee0c5b9bfcbfc8a33768c3d798bdec34654babee9367779ed5fcd4: Status 404 returned error can't find the container with id 3c2f2faa08ee0c5b9bfcbfc8a33768c3d798bdec34654babee9367779ed5fcd4
Jul 17 22:18:37.269803 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.269767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" event={"ID":"6ed224be-9547-4b5b-b069-4266c5cf68fa","Type":"ContainerStarted","Data":"9d91141c90a0d6a91b30dd1fe01a7301a62df3e9ee294d06ac6ad54ffc4141aa"}
Jul 17 22:18:37.270182 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.269838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" event={"ID":"6ed224be-9547-4b5b-b069-4266c5cf68fa","Type":"ContainerStarted","Data":"3c2f2faa08ee0c5b9bfcbfc8a33768c3d798bdec34654babee9367779ed5fcd4"}
Jul 17 22:18:37.271269 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.271248    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"
Jul 17 22:18:37.271385 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.271248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn" event={"ID":"25495407-47dd-4f75-96ec-5d3eb4e72f14","Type":"ContainerDied","Data":"aed1e7c60c1196ab706aad63fb387119ddc90c3e7b77941e4f5e2e7094896e30"}
Jul 17 22:18:37.271456 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.271400    2563 scope.go:117] "RemoveContainer" containerID="500b747c22dfe3fbf1b1d753a9f7916b2641aaad694768cc9d05cc9e23f315a8"
Jul 17 22:18:37.284194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.284155    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" podStartSLOduration=1.2841430649999999 podStartE2EDuration="1.284143065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:37.283467372 +0000 UTC m=+3081.195707689" watchObservedRunningTime="2026-07-17 22:18:37.284143065 +0000 UTC m=+3081.196383353"
Jul 17 22:18:37.326832 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.326792    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"]
Jul 17 22:18:37.328897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:37.328862    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-bd6zn"]
Jul 17 22:18:38.277759 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:38.277722    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerID="9d91141c90a0d6a91b30dd1fe01a7301a62df3e9ee294d06ac6ad54ffc4141aa" exitCode=6
Jul 17 22:18:38.278254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:38.277810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" event={"ID":"6ed224be-9547-4b5b-b069-4266c5cf68fa","Type":"ContainerDied","Data":"9d91141c90a0d6a91b30dd1fe01a7301a62df3e9ee294d06ac6ad54ffc4141aa"}
Jul 17 22:18:38.278254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:38.278019    2563 scope.go:117] "RemoveContainer" containerID="9d91141c90a0d6a91b30dd1fe01a7301a62df3e9ee294d06ac6ad54ffc4141aa"
Jul 17 22:18:38.602533 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:38.602450    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" path="/var/lib/kubelet/pods/25495407-47dd-4f75-96ec-5d3eb4e72f14/volumes"
Jul 17 22:18:39.285839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:39.285806    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerID="5999694f42e58bebc3c47efb2a1cedd978c4d5eef65679384c11637516e0598e" exitCode=6
Jul 17 22:18:39.286263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:39.285894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" event={"ID":"6ed224be-9547-4b5b-b069-4266c5cf68fa","Type":"ContainerDied","Data":"5999694f42e58bebc3c47efb2a1cedd978c4d5eef65679384c11637516e0598e"}
Jul 17 22:18:39.286263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:39.285949    2563 scope.go:117] "RemoveContainer" containerID="9d91141c90a0d6a91b30dd1fe01a7301a62df3e9ee294d06ac6ad54ffc4141aa"
Jul 17 22:18:39.286263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:39.286066    2563 scope.go:117] "RemoveContainer" containerID="5999694f42e58bebc3c47efb2a1cedd978c4d5eef65679384c11637516e0598e"
Jul 17 22:18:39.286409 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:39.286311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj_odh-ai-gateway-infra(6ed224be-9547-4b5b-b069-4266c5cf68fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa"
Jul 17 22:18:40.293044 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.293019    2563 scope.go:117] "RemoveContainer" containerID="5999694f42e58bebc3c47efb2a1cedd978c4d5eef65679384c11637516e0598e"
Jul 17 22:18:40.293437 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:40.293209    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj_odh-ai-gateway-infra(6ed224be-9547-4b5b-b069-4266c5cf68fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa"
Jul 17 22:18:40.360523 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.360482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"]
Jul 17 22:18:40.360988 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.360970    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:18:40.361038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.360990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:18:40.361038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.361011    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:18:40.361038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.361019    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:18:40.361148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.361091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:18:40.361148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.361099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:18:40.361148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.361108    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce6e8b5-2cd5-4e78-b50c-152afc3a6251" containerName="revoke-keys"
Jul 17 22:18:40.365315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.365296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:18:40.370117 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.370094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"]
Jul 17 22:18:40.478056 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.478023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01299b8d-2502-4585-b141-9e24acefd0f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z\" (UID: \"01299b8d-2502-4585-b141-9e24acefd0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:18:40.578766 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.578684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01299b8d-2502-4585-b141-9e24acefd0f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z\" (UID: \"01299b8d-2502-4585-b141-9e24acefd0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:18:40.579268 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.579249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01299b8d-2502-4585-b141-9e24acefd0f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z\" (UID: \"01299b8d-2502-4585-b141-9e24acefd0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:18:40.678047 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.678004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:18:40.819148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:40.819113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"]
Jul 17 22:18:40.821199 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:18:40.821169    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01299b8d_2502_4585_b141_9e24acefd0f0.slice/crio-44c1db5354c99f205acc631232818a0bfdaca2ecd671bbdf50114b42ffcf2bb3 WatchSource:0}: Error finding container 44c1db5354c99f205acc631232818a0bfdaca2ecd671bbdf50114b42ffcf2bb3: Status 404 returned error can't find the container with id 44c1db5354c99f205acc631232818a0bfdaca2ecd671bbdf50114b42ffcf2bb3
Jul 17 22:18:41.300870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:41.300831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" event={"ID":"01299b8d-2502-4585-b141-9e24acefd0f0","Type":"ContainerStarted","Data":"32912bab5e33b9f8d08668f9309a598e02399e4ca7dba71067a1d4e301b841c4"}
Jul 17 22:18:41.301273 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:41.300908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" event={"ID":"01299b8d-2502-4585-b141-9e24acefd0f0","Type":"ContainerStarted","Data":"44c1db5354c99f205acc631232818a0bfdaca2ecd671bbdf50114b42ffcf2bb3"}
Jul 17 22:18:41.315478 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:41.315433    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" podStartSLOduration=1.31542019 podStartE2EDuration="1.31542019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:41.314394042 +0000 UTC m=+3085.226634332" watchObservedRunningTime="2026-07-17 22:18:41.31542019 +0000 UTC m=+3085.227660479"
Jul 17 22:18:42.309112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:42.309077    2563 generic.go:356] "Generic (PLEG): container finished" podID="01299b8d-2502-4585-b141-9e24acefd0f0" containerID="32912bab5e33b9f8d08668f9309a598e02399e4ca7dba71067a1d4e301b841c4" exitCode=6
Jul 17 22:18:42.309535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:42.309166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" event={"ID":"01299b8d-2502-4585-b141-9e24acefd0f0","Type":"ContainerDied","Data":"32912bab5e33b9f8d08668f9309a598e02399e4ca7dba71067a1d4e301b841c4"}
Jul 17 22:18:42.309535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:42.309334    2563 scope.go:117] "RemoveContainer" containerID="32912bab5e33b9f8d08668f9309a598e02399e4ca7dba71067a1d4e301b841c4"
Jul 17 22:18:43.318766 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:43.318394    2563 generic.go:356] "Generic (PLEG): container finished" podID="01299b8d-2502-4585-b141-9e24acefd0f0" containerID="33b730d2e1da637a2f54cb3322f81d79ff7f0ada414f1eaf5f5dc30835a49d89" exitCode=6
Jul 17 22:18:43.318766 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:43.318499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" event={"ID":"01299b8d-2502-4585-b141-9e24acefd0f0","Type":"ContainerDied","Data":"33b730d2e1da637a2f54cb3322f81d79ff7f0ada414f1eaf5f5dc30835a49d89"}
Jul 17 22:18:43.318766 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:43.318561    2563 scope.go:117] "RemoveContainer" containerID="32912bab5e33b9f8d08668f9309a598e02399e4ca7dba71067a1d4e301b841c4"
Jul 17 22:18:43.319731 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:43.319707    2563 scope.go:117] "RemoveContainer" containerID="33b730d2e1da637a2f54cb3322f81d79ff7f0ada414f1eaf5f5dc30835a49d89"
Jul 17 22:18:43.320110 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:43.319956    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z_odh-ai-gateway-infra(01299b8d-2502-4585-b141-9e24acefd0f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" podUID="01299b8d-2502-4585-b141-9e24acefd0f0"
Jul 17 22:18:44.325644 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:44.325619    2563 scope.go:117] "RemoveContainer" containerID="33b730d2e1da637a2f54cb3322f81d79ff7f0ada414f1eaf5f5dc30835a49d89"
Jul 17 22:18:44.326021 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:44.325823    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z_odh-ai-gateway-infra(01299b8d-2502-4585-b141-9e24acefd0f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" podUID="01299b8d-2502-4585-b141-9e24acefd0f0"
Jul 17 22:18:51.597050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:51.597018    2563 scope.go:117] "RemoveContainer" containerID="5999694f42e58bebc3c47efb2a1cedd978c4d5eef65679384c11637516e0598e"
Jul 17 22:18:52.373091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:52.373058    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerID="fcf483108547bfd9273599f15206f09d3e47132f52c568c775b153c4c5bb0aad" exitCode=6
Jul 17 22:18:52.373257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:52.373130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" event={"ID":"6ed224be-9547-4b5b-b069-4266c5cf68fa","Type":"ContainerDied","Data":"fcf483108547bfd9273599f15206f09d3e47132f52c568c775b153c4c5bb0aad"}
Jul 17 22:18:52.373257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:52.373190    2563 scope.go:117] "RemoveContainer" containerID="5999694f42e58bebc3c47efb2a1cedd978c4d5eef65679384c11637516e0598e"
Jul 17 22:18:52.373551 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:52.373527    2563 scope.go:117] "RemoveContainer" containerID="fcf483108547bfd9273599f15206f09d3e47132f52c568c775b153c4c5bb0aad"
Jul 17 22:18:52.373814 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:18:52.373793    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj_odh-ai-gateway-infra(6ed224be-9547-4b5b-b069-4266c5cf68fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa"
Jul 17 22:18:53.399609 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:53.399564    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"]
Jul 17 22:18:53.551404 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:53.551381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:53.595421 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:53.595386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed224be-9547-4b5b-b069-4266c5cf68fa-maas-api-service-ca\") pod \"6ed224be-9547-4b5b-b069-4266c5cf68fa\" (UID: \"6ed224be-9547-4b5b-b069-4266c5cf68fa\") "
Jul 17 22:18:53.595760 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:53.595729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed224be-9547-4b5b-b069-4266c5cf68fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ed224be-9547-4b5b-b069-4266c5cf68fa" (UID: "6ed224be-9547-4b5b-b069-4266c5cf68fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:53.696010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:53.695977    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed224be-9547-4b5b-b069-4266c5cf68fa-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:18:54.386060 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:54.386029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"
Jul 17 22:18:54.386060 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:54.386049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj" event={"ID":"6ed224be-9547-4b5b-b069-4266c5cf68fa","Type":"ContainerDied","Data":"3c2f2faa08ee0c5b9bfcbfc8a33768c3d798bdec34654babee9367779ed5fcd4"}
Jul 17 22:18:54.386279 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:54.386115    2563 scope.go:117] "RemoveContainer" containerID="fcf483108547bfd9273599f15206f09d3e47132f52c568c775b153c4c5bb0aad"
Jul 17 22:18:54.407439 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:54.407411    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"]
Jul 17 22:18:54.412930 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:54.412905    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ph6tj"]
Jul 17 22:18:54.603111 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:54.603076    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" path="/var/lib/kubelet/pods/6ed224be-9547-4b5b-b069-4266c5cf68fa/volumes"
Jul 17 22:18:59.597381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:18:59.597352    2563 scope.go:117] "RemoveContainer" containerID="33b730d2e1da637a2f54cb3322f81d79ff7f0ada414f1eaf5f5dc30835a49d89"
Jul 17 22:19:00.419228 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:00.419199    2563 generic.go:356] "Generic (PLEG): container finished" podID="01299b8d-2502-4585-b141-9e24acefd0f0" containerID="219e83bd9cf1a5ccff2d100e4ec94b45a6d5948ace1cd6bc16f6fe8fc751efb0" exitCode=6
Jul 17 22:19:00.419408 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:00.419268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" event={"ID":"01299b8d-2502-4585-b141-9e24acefd0f0","Type":"ContainerDied","Data":"219e83bd9cf1a5ccff2d100e4ec94b45a6d5948ace1cd6bc16f6fe8fc751efb0"}
Jul 17 22:19:00.419408 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:00.419316    2563 scope.go:117] "RemoveContainer" containerID="33b730d2e1da637a2f54cb3322f81d79ff7f0ada414f1eaf5f5dc30835a49d89"
Jul 17 22:19:00.419670 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:00.419645    2563 scope.go:117] "RemoveContainer" containerID="219e83bd9cf1a5ccff2d100e4ec94b45a6d5948ace1cd6bc16f6fe8fc751efb0"
Jul 17 22:19:00.419910 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:00.419888    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z_odh-ai-gateway-infra(01299b8d-2502-4585-b141-9e24acefd0f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" podUID="01299b8d-2502-4585-b141-9e24acefd0f0"
Jul 17 22:19:01.443707 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:01.443673    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"]
Jul 17 22:19:01.611932 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:01.611907    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:19:01.770668 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:01.770564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01299b8d-2502-4585-b141-9e24acefd0f0-maas-api-service-ca\") pod \"01299b8d-2502-4585-b141-9e24acefd0f0\" (UID: \"01299b8d-2502-4585-b141-9e24acefd0f0\") "
Jul 17 22:19:01.770964 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:01.770942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01299b8d-2502-4585-b141-9e24acefd0f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01299b8d-2502-4585-b141-9e24acefd0f0" (UID: "01299b8d-2502-4585-b141-9e24acefd0f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:01.872035 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:01.872006    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01299b8d-2502-4585-b141-9e24acefd0f0-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:19:02.438701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:02.438670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"
Jul 17 22:19:02.438856 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:02.438669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z" event={"ID":"01299b8d-2502-4585-b141-9e24acefd0f0","Type":"ContainerDied","Data":"44c1db5354c99f205acc631232818a0bfdaca2ecd671bbdf50114b42ffcf2bb3"}
Jul 17 22:19:02.438856 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:02.438817    2563 scope.go:117] "RemoveContainer" containerID="219e83bd9cf1a5ccff2d100e4ec94b45a6d5948ace1cd6bc16f6fe8fc751efb0"
Jul 17 22:19:02.462348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:02.462316    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"]
Jul 17 22:19:02.467046 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:02.467024    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g4v4z"]
Jul 17 22:19:02.604014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:02.603983    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" path="/var/lib/kubelet/pods/01299b8d-2502-4585-b141-9e24acefd0f0/volumes"
Jul 17 22:19:03.798986 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.798947    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"]
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799491    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799521    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799532    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799541    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.799553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799560    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799717    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799731    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25495407-47dd-4f75-96ec-5d3eb4e72f14" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799742    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799750    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799760    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.799958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.799769    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:03.803975 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.803953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:03.806490 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.806471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:19:03.809709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.809688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"]
Jul 17 22:19:03.889694 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.889665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa23c84-0190-4a5f-a56e-507fa989267c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv\" (UID: \"6fa23c84-0190-4a5f-a56e-507fa989267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:03.990993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.990953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa23c84-0190-4a5f-a56e-507fa989267c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv\" (UID: \"6fa23c84-0190-4a5f-a56e-507fa989267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:03.991705 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:03.991682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa23c84-0190-4a5f-a56e-507fa989267c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv\" (UID: \"6fa23c84-0190-4a5f-a56e-507fa989267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:04.115200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:04.115123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:04.251662 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:04.251635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"]
Jul 17 22:19:04.253764 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:19:04.253732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa23c84_0190_4a5f_a56e_507fa989267c.slice/crio-da0bd53d5c2acb7b343dea5f462ef0f6b51a661ebcd9d0443c1dc6bc16528b31 WatchSource:0}: Error finding container da0bd53d5c2acb7b343dea5f462ef0f6b51a661ebcd9d0443c1dc6bc16528b31: Status 404 returned error can't find the container with id da0bd53d5c2acb7b343dea5f462ef0f6b51a661ebcd9d0443c1dc6bc16528b31
Jul 17 22:19:04.452463 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:04.452429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" event={"ID":"6fa23c84-0190-4a5f-a56e-507fa989267c","Type":"ContainerStarted","Data":"da0bd53d5c2acb7b343dea5f462ef0f6b51a661ebcd9d0443c1dc6bc16528b31"}
Jul 17 22:19:05.459036 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:05.459000    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerID="d03390bb22128b51fdfdd0e2a7ae360a734a64431083026eb683134fbb4c971d" exitCode=6
Jul 17 22:19:05.459416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:05.459083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" event={"ID":"6fa23c84-0190-4a5f-a56e-507fa989267c","Type":"ContainerDied","Data":"d03390bb22128b51fdfdd0e2a7ae360a734a64431083026eb683134fbb4c971d"}
Jul 17 22:19:05.459416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:05.459289    2563 scope.go:117] "RemoveContainer" containerID="d03390bb22128b51fdfdd0e2a7ae360a734a64431083026eb683134fbb4c971d"
Jul 17 22:19:06.466690 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:06.466655    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerID="8a74a36324407da4d63df47effa9ae4c526fa3b7c5b9afcce5fff35a082ae0e8" exitCode=6
Jul 17 22:19:06.467142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:06.466714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" event={"ID":"6fa23c84-0190-4a5f-a56e-507fa989267c","Type":"ContainerDied","Data":"8a74a36324407da4d63df47effa9ae4c526fa3b7c5b9afcce5fff35a082ae0e8"}
Jul 17 22:19:06.467142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:06.466754    2563 scope.go:117] "RemoveContainer" containerID="d03390bb22128b51fdfdd0e2a7ae360a734a64431083026eb683134fbb4c971d"
Jul 17 22:19:06.467142 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:06.466927    2563 scope.go:117] "RemoveContainer" containerID="8a74a36324407da4d63df47effa9ae4c526fa3b7c5b9afcce5fff35a082ae0e8"
Jul 17 22:19:06.467268 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:06.467140    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv_odh-ai-gateway-infra(6fa23c84-0190-4a5f-a56e-507fa989267c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c"
Jul 17 22:19:07.473365 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:07.473334    2563 scope.go:117] "RemoveContainer" containerID="8a74a36324407da4d63df47effa9ae4c526fa3b7c5b9afcce5fff35a082ae0e8"
Jul 17 22:19:07.473765 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:07.473551    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv_odh-ai-gateway-infra(6fa23c84-0190-4a5f-a56e-507fa989267c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c"
Jul 17 22:19:08.182104 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.182058    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"]
Jul 17 22:19:08.182520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.182502    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:08.182643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.182524    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed224be-9547-4b5b-b069-4266c5cf68fa" containerName="revoke-keys"
Jul 17 22:19:08.182643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.182550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:08.182643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.182559    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:08.182810 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.182690    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01299b8d-2502-4585-b141-9e24acefd0f0" containerName="revoke-keys"
Jul 17 22:19:08.187197 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.187172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:08.192346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.192314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"]
Jul 17 22:19:08.332645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.332579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2\" (UID: \"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:08.433898 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.433802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2\" (UID: \"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:08.434406 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.434386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2\" (UID: \"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:08.498680 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.498650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:08.638179 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:08.638132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"]
Jul 17 22:19:08.640427 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:19:08.640398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda168d5ec_80a8_49c8_bf0f_d1ebc5cd79b7.slice/crio-f254343b0f0b94cb57bcae4da099abb3874ae43fee510e4b8beb117243e9ac93 WatchSource:0}: Error finding container f254343b0f0b94cb57bcae4da099abb3874ae43fee510e4b8beb117243e9ac93: Status 404 returned error can't find the container with id f254343b0f0b94cb57bcae4da099abb3874ae43fee510e4b8beb117243e9ac93
Jul 17 22:19:09.485657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:09.485617    2563 generic.go:356] "Generic (PLEG): container finished" podID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerID="239eacc1d9a8c19f9f8f307aa5a1663450386f19b26813aa79f975b91a20e6fb" exitCode=6
Jul 17 22:19:09.485859 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:09.485720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" event={"ID":"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7","Type":"ContainerDied","Data":"239eacc1d9a8c19f9f8f307aa5a1663450386f19b26813aa79f975b91a20e6fb"}
Jul 17 22:19:09.485859 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:09.485781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" event={"ID":"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7","Type":"ContainerStarted","Data":"f254343b0f0b94cb57bcae4da099abb3874ae43fee510e4b8beb117243e9ac93"}
Jul 17 22:19:09.485954 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:09.485914    2563 scope.go:117] "RemoveContainer" containerID="239eacc1d9a8c19f9f8f307aa5a1663450386f19b26813aa79f975b91a20e6fb"
Jul 17 22:19:10.492846 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:10.492809    2563 generic.go:356] "Generic (PLEG): container finished" podID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerID="d295b75c1a1342f3f314278aa7e4a2b845d0636f1049b01785b6d7cb34b4204f" exitCode=6
Jul 17 22:19:10.493330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:10.492895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" event={"ID":"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7","Type":"ContainerDied","Data":"d295b75c1a1342f3f314278aa7e4a2b845d0636f1049b01785b6d7cb34b4204f"}
Jul 17 22:19:10.493330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:10.492949    2563 scope.go:117] "RemoveContainer" containerID="239eacc1d9a8c19f9f8f307aa5a1663450386f19b26813aa79f975b91a20e6fb"
Jul 17 22:19:10.493330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:10.493115    2563 scope.go:117] "RemoveContainer" containerID="d295b75c1a1342f3f314278aa7e4a2b845d0636f1049b01785b6d7cb34b4204f"
Jul 17 22:19:10.493487 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:10.493329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2_odh-ai-gateway-infra(a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7"
Jul 17 22:19:11.499918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:11.499890    2563 scope.go:117] "RemoveContainer" containerID="d295b75c1a1342f3f314278aa7e4a2b845d0636f1049b01785b6d7cb34b4204f"
Jul 17 22:19:11.500326 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:11.500064    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2_odh-ai-gateway-infra(a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7"
Jul 17 22:19:20.596780 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:20.596745    2563 scope.go:117] "RemoveContainer" containerID="8a74a36324407da4d63df47effa9ae4c526fa3b7c5b9afcce5fff35a082ae0e8"
Jul 17 22:19:21.555552 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:21.555519    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerID="a42ea67e88b8237687b2c5c0f96f3e17b63202cbf064bbf1519c2e405ae42e60" exitCode=6
Jul 17 22:19:21.555745 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:21.555592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" event={"ID":"6fa23c84-0190-4a5f-a56e-507fa989267c","Type":"ContainerDied","Data":"a42ea67e88b8237687b2c5c0f96f3e17b63202cbf064bbf1519c2e405ae42e60"}
Jul 17 22:19:21.555745 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:21.555668    2563 scope.go:117] "RemoveContainer" containerID="8a74a36324407da4d63df47effa9ae4c526fa3b7c5b9afcce5fff35a082ae0e8"
Jul 17 22:19:21.555958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:21.555930    2563 scope.go:117] "RemoveContainer" containerID="a42ea67e88b8237687b2c5c0f96f3e17b63202cbf064bbf1519c2e405ae42e60"
Jul 17 22:19:21.556191 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:21.556172    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv_odh-ai-gateway-infra(6fa23c84-0190-4a5f-a56e-507fa989267c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c"
Jul 17 22:19:22.580555 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:22.580521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"]
Jul 17 22:19:22.725606 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:22.725579    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:22.762915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:22.762883    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa23c84-0190-4a5f-a56e-507fa989267c-maas-api-service-ca\") pod \"6fa23c84-0190-4a5f-a56e-507fa989267c\" (UID: \"6fa23c84-0190-4a5f-a56e-507fa989267c\") "
Jul 17 22:19:22.763205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:22.763185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa23c84-0190-4a5f-a56e-507fa989267c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fa23c84-0190-4a5f-a56e-507fa989267c" (UID: "6fa23c84-0190-4a5f-a56e-507fa989267c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:22.864138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:22.864060    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa23c84-0190-4a5f-a56e-507fa989267c-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:19:23.569383 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:23.569354    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"
Jul 17 22:19:23.569383 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:23.569370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv" event={"ID":"6fa23c84-0190-4a5f-a56e-507fa989267c","Type":"ContainerDied","Data":"da0bd53d5c2acb7b343dea5f462ef0f6b51a661ebcd9d0443c1dc6bc16528b31"}
Jul 17 22:19:23.569674 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:23.569436    2563 scope.go:117] "RemoveContainer" containerID="a42ea67e88b8237687b2c5c0f96f3e17b63202cbf064bbf1519c2e405ae42e60"
Jul 17 22:19:23.592348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:23.592319    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"]
Jul 17 22:19:23.600107 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:23.600081    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-kmdnv"]
Jul 17 22:19:24.601910 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:24.601873    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" path="/var/lib/kubelet/pods/6fa23c84-0190-4a5f-a56e-507fa989267c/volumes"
Jul 17 22:19:25.597201 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:25.597174    2563 scope.go:117] "RemoveContainer" containerID="d295b75c1a1342f3f314278aa7e4a2b845d0636f1049b01785b6d7cb34b4204f"
Jul 17 22:19:26.082256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082224    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"]
Jul 17 22:19:26.082635 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082620    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:26.082679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:26.082679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082657    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:26.082679 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082662    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:26.082785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082725    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:26.082785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.082739    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:26.085908 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.085886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:26.094303 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.094274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"]
Jul 17 22:19:26.192898 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.192863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5087c3e-3c49-4dcb-84fc-36496603cce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4\" (UID: \"e5087c3e-3c49-4dcb-84fc-36496603cce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:26.294037 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.294006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5087c3e-3c49-4dcb-84fc-36496603cce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4\" (UID: \"e5087c3e-3c49-4dcb-84fc-36496603cce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:26.294640 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.294588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5087c3e-3c49-4dcb-84fc-36496603cce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4\" (UID: \"e5087c3e-3c49-4dcb-84fc-36496603cce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:26.400951 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.400862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:26.539097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.539071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"]
Jul 17 22:19:26.541857 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:19:26.541826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5087c3e_3c49_4dcb_84fc_36496603cce8.slice/crio-656cebb16dfbd832fe63bbf66aeec134b327ef03e4d203ba5c41280848835461 WatchSource:0}: Error finding container 656cebb16dfbd832fe63bbf66aeec134b327ef03e4d203ba5c41280848835461: Status 404 returned error can't find the container with id 656cebb16dfbd832fe63bbf66aeec134b327ef03e4d203ba5c41280848835461
Jul 17 22:19:26.586785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.586752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" event={"ID":"e5087c3e-3c49-4dcb-84fc-36496603cce8","Type":"ContainerStarted","Data":"656cebb16dfbd832fe63bbf66aeec134b327ef03e4d203ba5c41280848835461"}
Jul 17 22:19:26.588580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.588555    2563 generic.go:356] "Generic (PLEG): container finished" podID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerID="6af45a9b88f12b36d24eb219653e310ece5698cd856809727fc7bc6acd0ac81b" exitCode=6
Jul 17 22:19:26.588706 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.588643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" event={"ID":"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7","Type":"ContainerDied","Data":"6af45a9b88f12b36d24eb219653e310ece5698cd856809727fc7bc6acd0ac81b"}
Jul 17 22:19:26.588706 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.588704    2563 scope.go:117] "RemoveContainer" containerID="d295b75c1a1342f3f314278aa7e4a2b845d0636f1049b01785b6d7cb34b4204f"
Jul 17 22:19:26.589064 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:26.589050    2563 scope.go:117] "RemoveContainer" containerID="6af45a9b88f12b36d24eb219653e310ece5698cd856809727fc7bc6acd0ac81b"
Jul 17 22:19:26.589292 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:26.589270    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2_odh-ai-gateway-infra(a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7"
Jul 17 22:19:27.595492 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.595403    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerID="1af7fadd0ffd650d28c57b75a49d99b1eb374d5754180b6721183627e194c2b6" exitCode=6
Jul 17 22:19:27.595941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.595497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" event={"ID":"e5087c3e-3c49-4dcb-84fc-36496603cce8","Type":"ContainerDied","Data":"1af7fadd0ffd650d28c57b75a49d99b1eb374d5754180b6721183627e194c2b6"}
Jul 17 22:19:27.595941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.595722    2563 scope.go:117] "RemoveContainer" containerID="1af7fadd0ffd650d28c57b75a49d99b1eb374d5754180b6721183627e194c2b6"
Jul 17 22:19:27.621846 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.621818    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"]
Jul 17 22:19:27.787411 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.787386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:27.906191 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.906106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7-maas-api-service-ca\") pod \"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7\" (UID: \"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7\") "
Jul 17 22:19:27.906497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:27.906470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" (UID: "a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:28.007645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.007587    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:19:28.604757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.604723    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerID="2a3346b21cffd54767f8b80c064e15e86829b66c1bb83dce44ffdf2816beed3f" exitCode=6
Jul 17 22:19:28.605205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.604806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" event={"ID":"e5087c3e-3c49-4dcb-84fc-36496603cce8","Type":"ContainerDied","Data":"2a3346b21cffd54767f8b80c064e15e86829b66c1bb83dce44ffdf2816beed3f"}
Jul 17 22:19:28.605205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.604867    2563 scope.go:117] "RemoveContainer" containerID="1af7fadd0ffd650d28c57b75a49d99b1eb374d5754180b6721183627e194c2b6"
Jul 17 22:19:28.605205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.605050    2563 scope.go:117] "RemoveContainer" containerID="2a3346b21cffd54767f8b80c064e15e86829b66c1bb83dce44ffdf2816beed3f"
Jul 17 22:19:28.605397 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:28.605329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4_odh-ai-gateway-infra(e5087c3e-3c49-4dcb-84fc-36496603cce8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8"
Jul 17 22:19:28.606264 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.606246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"
Jul 17 22:19:28.606264 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.606253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2" event={"ID":"a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7","Type":"ContainerDied","Data":"f254343b0f0b94cb57bcae4da099abb3874ae43fee510e4b8beb117243e9ac93"}
Jul 17 22:19:28.617897 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.617710    2563 scope.go:117] "RemoveContainer" containerID="6af45a9b88f12b36d24eb219653e310ece5698cd856809727fc7bc6acd0ac81b"
Jul 17 22:19:28.632919 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.632897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"]
Jul 17 22:19:28.636545 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:28.636523    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-5gcf2"]
Jul 17 22:19:29.613726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:29.613694    2563 scope.go:117] "RemoveContainer" containerID="2a3346b21cffd54767f8b80c064e15e86829b66c1bb83dce44ffdf2816beed3f"
Jul 17 22:19:29.614177 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:29.613904    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4_odh-ai-gateway-infra(e5087c3e-3c49-4dcb-84fc-36496603cce8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8"
Jul 17 22:19:30.603093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:30.603061    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" path="/var/lib/kubelet/pods/a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7/volumes"
Jul 17 22:19:40.407114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407025    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"]
Jul 17 22:19:40.407709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.407709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.407709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407655    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:40.407709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407665    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:40.407709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407678    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.407709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407688    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.408058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407808    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa23c84-0190-4a5f-a56e-507fa989267c" containerName="revoke-keys"
Jul 17 22:19:40.408058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407820    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.408058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407831    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.408058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.407842    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:40.412392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.412372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:19:40.417752 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.417727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"]
Jul 17 22:19:40.516196 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.516149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24279d89-1682-48dc-97cb-962b7d8f47d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f\" (UID: \"24279d89-1682-48dc-97cb-962b7d8f47d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:19:40.596773 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.596744    2563 scope.go:117] "RemoveContainer" containerID="2a3346b21cffd54767f8b80c064e15e86829b66c1bb83dce44ffdf2816beed3f"
Jul 17 22:19:40.617315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.617289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24279d89-1682-48dc-97cb-962b7d8f47d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f\" (UID: \"24279d89-1682-48dc-97cb-962b7d8f47d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:19:40.618073 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.618053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24279d89-1682-48dc-97cb-962b7d8f47d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f\" (UID: \"24279d89-1682-48dc-97cb-962b7d8f47d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:19:40.724408 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.724374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:19:40.864460 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:40.864427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"]
Jul 17 22:19:40.866101 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:19:40.865990    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24279d89_1682_48dc_97cb_962b7d8f47d2.slice/crio-4281f992bb4d6bab63210369f6b3a15aee5fbabf648b985f30b446428706321e WatchSource:0}: Error finding container 4281f992bb4d6bab63210369f6b3a15aee5fbabf648b985f30b446428706321e: Status 404 returned error can't find the container with id 4281f992bb4d6bab63210369f6b3a15aee5fbabf648b985f30b446428706321e
Jul 17 22:19:41.678563 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.678531    2563 generic.go:356] "Generic (PLEG): container finished" podID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerID="9f3c1e8e17adf97998e246e9724b3670cf58322bf2c0b116628009ed55aec7ed" exitCode=6
Jul 17 22:19:41.679005 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.678625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" event={"ID":"e5087c3e-3c49-4dcb-84fc-36496603cce8","Type":"ContainerDied","Data":"9f3c1e8e17adf97998e246e9724b3670cf58322bf2c0b116628009ed55aec7ed"}
Jul 17 22:19:41.679005 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.678692    2563 scope.go:117] "RemoveContainer" containerID="2a3346b21cffd54767f8b80c064e15e86829b66c1bb83dce44ffdf2816beed3f"
Jul 17 22:19:41.679005 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.678996    2563 scope.go:117] "RemoveContainer" containerID="9f3c1e8e17adf97998e246e9724b3670cf58322bf2c0b116628009ed55aec7ed"
Jul 17 22:19:41.679249 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:41.679224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4_odh-ai-gateway-infra(e5087c3e-3c49-4dcb-84fc-36496603cce8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8"
Jul 17 22:19:41.680336 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.680315    2563 generic.go:356] "Generic (PLEG): container finished" podID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerID="0ef1793bfb04b5470e8eaff50adc7b7a8e2da9c8eed922fca493f1db5314d170" exitCode=6
Jul 17 22:19:41.680415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.680395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" event={"ID":"24279d89-1682-48dc-97cb-962b7d8f47d2","Type":"ContainerDied","Data":"0ef1793bfb04b5470e8eaff50adc7b7a8e2da9c8eed922fca493f1db5314d170"}
Jul 17 22:19:41.680459 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.680433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" event={"ID":"24279d89-1682-48dc-97cb-962b7d8f47d2","Type":"ContainerStarted","Data":"4281f992bb4d6bab63210369f6b3a15aee5fbabf648b985f30b446428706321e"}
Jul 17 22:19:41.680575 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:41.680561    2563 scope.go:117] "RemoveContainer" containerID="0ef1793bfb04b5470e8eaff50adc7b7a8e2da9c8eed922fca493f1db5314d170"
Jul 17 22:19:42.688721 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.688688    2563 generic.go:356] "Generic (PLEG): container finished" podID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerID="c8d0fd84fe44a7f8b1351cc4b07e08c4d9c63e35e90e78ddd23e73a884856eee" exitCode=6
Jul 17 22:19:42.689156 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.688772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" event={"ID":"24279d89-1682-48dc-97cb-962b7d8f47d2","Type":"ContainerDied","Data":"c8d0fd84fe44a7f8b1351cc4b07e08c4d9c63e35e90e78ddd23e73a884856eee"}
Jul 17 22:19:42.689156 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.688828    2563 scope.go:117] "RemoveContainer" containerID="0ef1793bfb04b5470e8eaff50adc7b7a8e2da9c8eed922fca493f1db5314d170"
Jul 17 22:19:42.689156 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.688991    2563 scope.go:117] "RemoveContainer" containerID="c8d0fd84fe44a7f8b1351cc4b07e08c4d9c63e35e90e78ddd23e73a884856eee"
Jul 17 22:19:42.689340 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:42.689211    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f_odh-ai-gateway-infra(24279d89-1682-48dc-97cb-962b7d8f47d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2"
Jul 17 22:19:42.703464 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.703441    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"]
Jul 17 22:19:42.833792 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.833757    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:42.937695 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.937660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5087c3e-3c49-4dcb-84fc-36496603cce8-maas-api-service-ca\") pod \"e5087c3e-3c49-4dcb-84fc-36496603cce8\" (UID: \"e5087c3e-3c49-4dcb-84fc-36496603cce8\") "
Jul 17 22:19:42.938054 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:42.938033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5087c3e-3c49-4dcb-84fc-36496603cce8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5087c3e-3c49-4dcb-84fc-36496603cce8" (UID: "e5087c3e-3c49-4dcb-84fc-36496603cce8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:43.038749 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.038662    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5087c3e-3c49-4dcb-84fc-36496603cce8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:19:43.695006 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.694981    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"
Jul 17 22:19:43.695006 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.694995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4" event={"ID":"e5087c3e-3c49-4dcb-84fc-36496603cce8","Type":"ContainerDied","Data":"656cebb16dfbd832fe63bbf66aeec134b327ef03e4d203ba5c41280848835461"}
Jul 17 22:19:43.695530 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.695066    2563 scope.go:117] "RemoveContainer" containerID="9f3c1e8e17adf97998e246e9724b3670cf58322bf2c0b116628009ed55aec7ed"
Jul 17 22:19:43.696884 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.696870    2563 scope.go:117] "RemoveContainer" containerID="c8d0fd84fe44a7f8b1351cc4b07e08c4d9c63e35e90e78ddd23e73a884856eee"
Jul 17 22:19:43.697072 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:43.697053    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f_odh-ai-gateway-infra(24279d89-1682-48dc-97cb-962b7d8f47d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2"
Jul 17 22:19:43.729334 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.729304    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"]
Jul 17 22:19:43.732565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:43.732538    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-5dvk4"]
Jul 17 22:19:44.603320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:44.603286    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" path="/var/lib/kubelet/pods/e5087c3e-3c49-4dcb-84fc-36496603cce8/volumes"
Jul 17 22:19:46.700614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:46.700568    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"]
Jul 17 22:19:47.695625 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:47.695562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"]
Jul 17 22:19:53.210842 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.210809    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"]
Jul 17 22:19:53.211389 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211370    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:19:53.211467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211392    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:19:53.211467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211409    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:53.211467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211417    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a168d5ec-80a8-49c8-bf0f-d1ebc5cd79b7" containerName="revoke-keys"
Jul 17 22:19:53.211467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:19:53.211467 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:19:53.211750 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211568    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:19:53.211750 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.211583    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:19:53.216163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.216140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.218557 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.218536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"sub-test-model-8e6a3a-kserve-self-signed-certs\""
Jul 17 22:19:53.222680 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.222656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"]
Jul 17 22:19:53.335417 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.335373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-dshm\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.335417 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.335422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-home\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.335728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.335448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-model-cache\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.335728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.335507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae8b611-14d9-48ff-8651-4358f85d03cf-tls-certs\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.335728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.335553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdlx\" (UniqueName: \"kubernetes.io/projected/5ae8b611-14d9-48ff-8651-4358f85d03cf-kube-api-access-9gdlx\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.335728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.335580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-tmp-dir\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436355 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-dshm\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436355 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-home\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436641 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-model-cache\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436641 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae8b611-14d9-48ff-8651-4358f85d03cf-tls-certs\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436641 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdlx\" (UniqueName: \"kubernetes.io/projected/5ae8b611-14d9-48ff-8651-4358f85d03cf-kube-api-access-9gdlx\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436641 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-tmp-dir\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436875 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-home\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436918 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-model-cache\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.436954 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.436920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-tmp-dir\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.438805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.438783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-dshm\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.439155 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.439139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae8b611-14d9-48ff-8651-4358f85d03cf-tls-certs\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.444646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.444622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdlx\" (UniqueName: \"kubernetes.io/projected/5ae8b611-14d9-48ff-8651-4358f85d03cf-kube-api-access-9gdlx\") pod \"sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") " pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.527318 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.527231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:19:53.667298 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.667273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"]
Jul 17 22:19:53.670087 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:19:53.670049    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae8b611_14d9_48ff_8651_4358f85d03cf.slice/crio-8bcc2cda1c656d74e6582a0ff00744345e01734785889d960c8a0b8a8f1ef7e8 WatchSource:0}: Error finding container 8bcc2cda1c656d74e6582a0ff00744345e01734785889d960c8a0b8a8f1ef7e8: Status 404 returned error can't find the container with id 8bcc2cda1c656d74e6582a0ff00744345e01734785889d960c8a0b8a8f1ef7e8
Jul 17 22:19:53.758065 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.758034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5" event={"ID":"5ae8b611-14d9-48ff-8651-4358f85d03cf","Type":"ContainerStarted","Data":"59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827"}
Jul 17 22:19:53.758218 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.758075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5" event={"ID":"5ae8b611-14d9-48ff-8651-4358f85d03cf","Type":"ContainerStarted","Data":"8bcc2cda1c656d74e6582a0ff00744345e01734785889d960c8a0b8a8f1ef7e8"}
Jul 17 22:19:53.776994 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:53.776944    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5" podStartSLOduration=0.776927099 podStartE2EDuration="776.927099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:19:53.775667781 +0000 UTC m=+3157.687908075" watchObservedRunningTime="2026-07-17 22:19:53.776927099 +0000 UTC m=+3157.689167397"
Jul 17 22:19:58.597313 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:58.597280    2563 scope.go:117] "RemoveContainer" containerID="c8d0fd84fe44a7f8b1351cc4b07e08c4d9c63e35e90e78ddd23e73a884856eee"
Jul 17 22:19:59.790385 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:59.790352    2563 generic.go:356] "Generic (PLEG): container finished" podID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerID="9a734833433cc4889da5aa59f79878afbe93d808b1ff53bc2fadaa7e0c1bdc4a" exitCode=6
Jul 17 22:19:59.790796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:59.790424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" event={"ID":"24279d89-1682-48dc-97cb-962b7d8f47d2","Type":"ContainerDied","Data":"9a734833433cc4889da5aa59f79878afbe93d808b1ff53bc2fadaa7e0c1bdc4a"}
Jul 17 22:19:59.790796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:59.790510    2563 scope.go:117] "RemoveContainer" containerID="c8d0fd84fe44a7f8b1351cc4b07e08c4d9c63e35e90e78ddd23e73a884856eee"
Jul 17 22:19:59.790889 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:19:59.790842    2563 scope.go:117] "RemoveContainer" containerID="9a734833433cc4889da5aa59f79878afbe93d808b1ff53bc2fadaa7e0c1bdc4a"
Jul 17 22:19:59.791090 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:19:59.791069    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f_odh-ai-gateway-infra(24279d89-1682-48dc-97cb-962b7d8f47d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2"
Jul 17 22:20:00.816572 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:00.816540    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"]
Jul 17 22:20:00.956322 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:00.956297    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:20:01.113864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.113776    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24279d89-1682-48dc-97cb-962b7d8f47d2-maas-api-service-ca\") pod \"24279d89-1682-48dc-97cb-962b7d8f47d2\" (UID: \"24279d89-1682-48dc-97cb-962b7d8f47d2\") "
Jul 17 22:20:01.114169 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.114140    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24279d89-1682-48dc-97cb-962b7d8f47d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24279d89-1682-48dc-97cb-962b7d8f47d2" (UID: "24279d89-1682-48dc-97cb-962b7d8f47d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:01.215434 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.215398    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24279d89-1682-48dc-97cb-962b7d8f47d2-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:01.721447 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.721393    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk" podUID="ae2f15f7-ee18-4863-a8fb-e8430f6f9763" containerName="main" containerID="cri-o://e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887" gracePeriod=45
Jul 17 22:20:01.805110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.805070    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"
Jul 17 22:20:01.805110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.805088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f" event={"ID":"24279d89-1682-48dc-97cb-962b7d8f47d2","Type":"ContainerDied","Data":"4281f992bb4d6bab63210369f6b3a15aee5fbabf648b985f30b446428706321e"}
Jul 17 22:20:01.805307 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.805162    2563 scope.go:117] "RemoveContainer" containerID="9a734833433cc4889da5aa59f79878afbe93d808b1ff53bc2fadaa7e0c1bdc4a"
Jul 17 22:20:01.838337 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.838301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"]
Jul 17 22:20:01.842153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.842127    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-6vj8f"]
Jul 17 22:20:01.983327 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:01.983263    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:20:02.022916 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.022881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-home\") pod \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") "
Jul 17 22:20:02.022916 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.022914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-model-cache\") pod \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") "
Jul 17 22:20:02.023164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.022940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tls-certs\") pod \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") "
Jul 17 22:20:02.023164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.022978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tmp-dir\") pod \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") "
Jul 17 22:20:02.023164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.023022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-dshm\") pod \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") "
Jul 17 22:20:02.023339 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.023166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae2f15f7-ee18-4863-a8fb-e8430f6f9763" (UID: "ae2f15f7-ee18-4863-a8fb-e8430f6f9763"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:02.023339 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.023195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-home" (OuterVolumeSpecName: "home") pod "ae2f15f7-ee18-4863-a8fb-e8430f6f9763" (UID: "ae2f15f7-ee18-4863-a8fb-e8430f6f9763"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:02.023339 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.023259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae2f15f7-ee18-4863-a8fb-e8430f6f9763" (UID: "ae2f15f7-ee18-4863-a8fb-e8430f6f9763"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:02.023339 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.023338    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:02.023566 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.023351    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:02.025390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.025357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae2f15f7-ee18-4863-a8fb-e8430f6f9763" (UID: "ae2f15f7-ee18-4863-a8fb-e8430f6f9763"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:20:02.025505 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.025407    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-dshm" (OuterVolumeSpecName: "dshm") pod "ae2f15f7-ee18-4863-a8fb-e8430f6f9763" (UID: "ae2f15f7-ee18-4863-a8fb-e8430f6f9763"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:02.123859 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.123825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l27z2\" (UniqueName: \"kubernetes.io/projected/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-kube-api-access-l27z2\") pod \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\" (UID: \"ae2f15f7-ee18-4863-a8fb-e8430f6f9763\") "
Jul 17 22:20:02.124066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.124053    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:02.124066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.124067    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:02.124211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.124074    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:02.126034 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.126008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-kube-api-access-l27z2" (OuterVolumeSpecName: "kube-api-access-l27z2") pod "ae2f15f7-ee18-4863-a8fb-e8430f6f9763" (UID: "ae2f15f7-ee18-4863-a8fb-e8430f6f9763"). InnerVolumeSpecName "kube-api-access-l27z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:20:02.224558 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.224528    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27z2\" (UniqueName: \"kubernetes.io/projected/ae2f15f7-ee18-4863-a8fb-e8430f6f9763-kube-api-access-l27z2\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:02.602010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.601976    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" path="/var/lib/kubelet/pods/24279d89-1682-48dc-97cb-962b7d8f47d2/volumes"
Jul 17 22:20:02.717248 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.717207    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5" podUID="a943a051-83e5-485f-ac24-1a5c0b8b463e" containerName="main" containerID="cri-o://f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64" gracePeriod=45
Jul 17 22:20:02.811520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.811491    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae2f15f7-ee18-4863-a8fb-e8430f6f9763" containerID="e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887" exitCode=0
Jul 17 22:20:02.811709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.811534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"
Jul 17 22:20:02.811709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.811585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk" event={"ID":"ae2f15f7-ee18-4863-a8fb-e8430f6f9763","Type":"ContainerDied","Data":"e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887"}
Jul 17 22:20:02.811709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.811644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk" event={"ID":"ae2f15f7-ee18-4863-a8fb-e8430f6f9763","Type":"ContainerDied","Data":"2e581dde39aac6946bb9efa6726337741b4f9935577c07512c472a1e54e904db"}
Jul 17 22:20:02.811709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.811677    2563 scope.go:117] "RemoveContainer" containerID="e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887"
Jul 17 22:20:02.828791 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.828766    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"]
Jul 17 22:20:02.830582 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.830566    2563 scope.go:117] "RemoveContainer" containerID="e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887"
Jul 17 22:20:02.830895 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:02.830876    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887\": container with ID starting with e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887 not found: ID does not exist" containerID="e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887"
Jul 17 22:20:02.830981 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.830918    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887"} err="failed to get container status \"e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887\": rpc error: code = NotFound desc = could not find container \"e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887\": container with ID starting with e66db04c34fe3106701ee1344806840cff13de86f02516ab1f5504731f369887 not found: ID does not exist"
Jul 17 22:20:02.833869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.833848    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/auth-test-model-8e6a3a-kserve-97d4cf95-8zdgk"]
Jul 17 22:20:02.965610 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:02.965575    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:20:03.031349 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-tmp-dir\") pod \"a943a051-83e5-485f-ac24-1a5c0b8b463e\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") "
Jul 17 22:20:03.031526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031359    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-dshm\") pod \"a943a051-83e5-485f-ac24-1a5c0b8b463e\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") "
Jul 17 22:20:03.031526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a943a051-83e5-485f-ac24-1a5c0b8b463e-tls-certs\") pod \"a943a051-83e5-485f-ac24-1a5c0b8b463e\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") "
Jul 17 22:20:03.031526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-model-cache\") pod \"a943a051-83e5-485f-ac24-1a5c0b8b463e\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") "
Jul 17 22:20:03.031526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kd96\" (UniqueName: \"kubernetes.io/projected/a943a051-83e5-485f-ac24-1a5c0b8b463e-kube-api-access-5kd96\") pod \"a943a051-83e5-485f-ac24-1a5c0b8b463e\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") "
Jul 17 22:20:03.031790 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a943a051-83e5-485f-ac24-1a5c0b8b463e" (UID: "a943a051-83e5-485f-ac24-1a5c0b8b463e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:03.031790 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.031758    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a943a051-83e5-485f-ac24-1a5c0b8b463e" (UID: "a943a051-83e5-485f-ac24-1a5c0b8b463e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:03.033907 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.033878    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a943a051-83e5-485f-ac24-1a5c0b8b463e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a943a051-83e5-485f-ac24-1a5c0b8b463e" (UID: "a943a051-83e5-485f-ac24-1a5c0b8b463e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:20:03.033907 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.033882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a943a051-83e5-485f-ac24-1a5c0b8b463e-kube-api-access-5kd96" (OuterVolumeSpecName: "kube-api-access-5kd96") pod "a943a051-83e5-485f-ac24-1a5c0b8b463e" (UID: "a943a051-83e5-485f-ac24-1a5c0b8b463e"). InnerVolumeSpecName "kube-api-access-5kd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:20:03.034057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.033877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-dshm" (OuterVolumeSpecName: "dshm") pod "a943a051-83e5-485f-ac24-1a5c0b8b463e" (UID: "a943a051-83e5-485f-ac24-1a5c0b8b463e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:03.132730 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.132686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-home\") pod \"a943a051-83e5-485f-ac24-1a5c0b8b463e\" (UID: \"a943a051-83e5-485f-ac24-1a5c0b8b463e\") "
Jul 17 22:20:03.132960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.132945    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a943a051-83e5-485f-ac24-1a5c0b8b463e-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:03.133016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.132952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-home" (OuterVolumeSpecName: "home") pod "a943a051-83e5-485f-ac24-1a5c0b8b463e" (UID: "a943a051-83e5-485f-ac24-1a5c0b8b463e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:20:03.133016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.132963    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:03.133016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.133000    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kd96\" (UniqueName: \"kubernetes.io/projected/a943a051-83e5-485f-ac24-1a5c0b8b463e-kube-api-access-5kd96\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:03.133016 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.133011    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:03.133143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.133020    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:03.234473 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.234433    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a943a051-83e5-485f-ac24-1a5c0b8b463e-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:03.527687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.527583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:20:03.527687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.527670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:20:03.540614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.540570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:20:03.820965 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.820875    2563 generic.go:356] "Generic (PLEG): container finished" podID="a943a051-83e5-485f-ac24-1a5c0b8b463e" containerID="f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64" exitCode=0
Jul 17 22:20:03.820965 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.820921    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"
Jul 17 22:20:03.821187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.820964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5" event={"ID":"a943a051-83e5-485f-ac24-1a5c0b8b463e","Type":"ContainerDied","Data":"f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64"}
Jul 17 22:20:03.821187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.821007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5" event={"ID":"a943a051-83e5-485f-ac24-1a5c0b8b463e","Type":"ContainerDied","Data":"2d04e5ce139a48bc214990455d21e758c2ed860ab15f1fe5242c5357201db6e5"}
Jul 17 22:20:03.821187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.821030    2563 scope.go:117] "RemoveContainer" containerID="f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64"
Jul 17 22:20:03.837013 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.836986    2563 scope.go:117] "RemoveContainer" containerID="f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64"
Jul 17 22:20:03.837765 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:03.837704    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64\": container with ID starting with f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64 not found: ID does not exist" containerID="f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64"
Jul 17 22:20:03.838008 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.837802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64"} err="failed to get container status \"f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64\": rpc error: code = NotFound desc = could not find container \"f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64\": container with ID starting with f66b0e1f35a725f7c7b82611b28b82fc74e91d88d78f5add24f87e007ba00b64 not found: ID does not exist"
Jul 17 22:20:03.838645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.838624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:20:03.853845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.853792    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"]
Jul 17 22:20:03.856270 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:03.856250    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/auth-test-model-5401e5-kserve-785f76d9d7-qgrp5"]
Jul 17 22:20:04.606011 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:04.605979    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a943a051-83e5-485f-ac24-1a5c0b8b463e" path="/var/lib/kubelet/pods/a943a051-83e5-485f-ac24-1a5c0b8b463e/volumes"
Jul 17 22:20:04.606389 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:04.606349    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2f15f7-ee18-4863-a8fb-e8430f6f9763" path="/var/lib/kubelet/pods/ae2f15f7-ee18-4863-a8fb-e8430f6f9763/volumes"
Jul 17 22:20:08.229963 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.229934    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"]
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a943a051-83e5-485f-ac24-1a5c0b8b463e" containerName="main"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a943a051-83e5-485f-ac24-1a5c0b8b463e" containerName="main"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230301    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230307    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230323    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230341    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230351    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2f15f7-ee18-4863-a8fb-e8430f6f9763" containerName="main"
Jul 17 22:20:08.230356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230356    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2f15f7-ee18-4863-a8fb-e8430f6f9763" containerName="main"
Jul 17 22:20:08.230782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:08.230782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230424    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2f15f7-ee18-4863-a8fb-e8430f6f9763" containerName="main"
Jul 17 22:20:08.230782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230431    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5087c3e-3c49-4dcb-84fc-36496603cce8" containerName="revoke-keys"
Jul 17 22:20:08.230782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230438    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:08.230782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.230445    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a943a051-83e5-485f-ac24-1a5c0b8b463e" containerName="main"
Jul 17 22:20:08.234779 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.234755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:08.237495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.237475    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:20:08.242356 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.242327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"]
Jul 17 22:20:08.279989 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.279950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587e7da-f94e-4488-8780-7d5f50148336-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj\" (UID: \"3587e7da-f94e-4488-8780-7d5f50148336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:08.380806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.380762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587e7da-f94e-4488-8780-7d5f50148336-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj\" (UID: \"3587e7da-f94e-4488-8780-7d5f50148336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:08.381407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.381385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587e7da-f94e-4488-8780-7d5f50148336-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj\" (UID: \"3587e7da-f94e-4488-8780-7d5f50148336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:08.549022 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.548937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:08.698791 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.698765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"]
Jul 17 22:20:08.699370 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:20:08.699341    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3587e7da_f94e_4488_8780_7d5f50148336.slice/crio-9802513b8a8257828edbea4bf1ab0c0dc9177b0ab09983a922fae2a43f08bcb6 WatchSource:0}: Error finding container 9802513b8a8257828edbea4bf1ab0c0dc9177b0ab09983a922fae2a43f08bcb6: Status 404 returned error can't find the container with id 9802513b8a8257828edbea4bf1ab0c0dc9177b0ab09983a922fae2a43f08bcb6
Jul 17 22:20:08.852114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:08.852033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" event={"ID":"3587e7da-f94e-4488-8780-7d5f50148336","Type":"ContainerStarted","Data":"9802513b8a8257828edbea4bf1ab0c0dc9177b0ab09983a922fae2a43f08bcb6"}
Jul 17 22:20:09.859269 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:09.859230    2563 generic.go:356] "Generic (PLEG): container finished" podID="3587e7da-f94e-4488-8780-7d5f50148336" containerID="00d9b371e570ca4d2a8fb7a22608efe920c1733ba2709b7d4664f3c4f98ca3bf" exitCode=6
Jul 17 22:20:09.859696 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:09.859300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" event={"ID":"3587e7da-f94e-4488-8780-7d5f50148336","Type":"ContainerDied","Data":"00d9b371e570ca4d2a8fb7a22608efe920c1733ba2709b7d4664f3c4f98ca3bf"}
Jul 17 22:20:09.859696 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:09.859449    2563 scope.go:117] "RemoveContainer" containerID="00d9b371e570ca4d2a8fb7a22608efe920c1733ba2709b7d4664f3c4f98ca3bf"
Jul 17 22:20:10.867219 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:10.867128    2563 generic.go:356] "Generic (PLEG): container finished" podID="3587e7da-f94e-4488-8780-7d5f50148336" containerID="6620faa8b69d42a06a45d28e73fdafd8b38d9bbf341a7ce5e34282ff52540a2d" exitCode=6
Jul 17 22:20:10.867219 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:10.867193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" event={"ID":"3587e7da-f94e-4488-8780-7d5f50148336","Type":"ContainerDied","Data":"6620faa8b69d42a06a45d28e73fdafd8b38d9bbf341a7ce5e34282ff52540a2d"}
Jul 17 22:20:10.867732 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:10.867234    2563 scope.go:117] "RemoveContainer" containerID="00d9b371e570ca4d2a8fb7a22608efe920c1733ba2709b7d4664f3c4f98ca3bf"
Jul 17 22:20:10.867732 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:10.867437    2563 scope.go:117] "RemoveContainer" containerID="6620faa8b69d42a06a45d28e73fdafd8b38d9bbf341a7ce5e34282ff52540a2d"
Jul 17 22:20:10.867732 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:10.867665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj_odh-ai-gateway-infra(3587e7da-f94e-4488-8780-7d5f50148336)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" podUID="3587e7da-f94e-4488-8780-7d5f50148336"
Jul 17 22:20:11.874420 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:11.874391    2563 scope.go:117] "RemoveContainer" containerID="6620faa8b69d42a06a45d28e73fdafd8b38d9bbf341a7ce5e34282ff52540a2d"
Jul 17 22:20:11.874915 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:11.874618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj_odh-ai-gateway-infra(3587e7da-f94e-4488-8780-7d5f50148336)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" podUID="3587e7da-f94e-4488-8780-7d5f50148336"
Jul 17 22:20:16.151653 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.151616    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"]
Jul 17 22:20:16.152066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.152020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:16.152066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.152031    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:16.152159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.152095    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24279d89-1682-48dc-97cb-962b7d8f47d2" containerName="revoke-keys"
Jul 17 22:20:16.156363 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.156345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:16.164579 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.164550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"]
Jul 17 22:20:16.251066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.251024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b12311-e11e-4640-a6e2-07ed6b88785a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4\" (UID: \"86b12311-e11e-4640-a6e2-07ed6b88785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:16.352327 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.352279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b12311-e11e-4640-a6e2-07ed6b88785a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4\" (UID: \"86b12311-e11e-4640-a6e2-07ed6b88785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:16.352934 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.352916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b12311-e11e-4640-a6e2-07ed6b88785a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4\" (UID: \"86b12311-e11e-4640-a6e2-07ed6b88785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:16.468868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.468822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:16.609977 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.609914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"]
Jul 17 22:20:16.612208 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:20:16.612178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b12311_e11e_4640_a6e2_07ed6b88785a.slice/crio-52d0c5fa432bfc061b266d415c4fa2ebab892ba4a3f0bc89d8b96686ef81b3b0 WatchSource:0}: Error finding container 52d0c5fa432bfc061b266d415c4fa2ebab892ba4a3f0bc89d8b96686ef81b3b0: Status 404 returned error can't find the container with id 52d0c5fa432bfc061b266d415c4fa2ebab892ba4a3f0bc89d8b96686ef81b3b0
Jul 17 22:20:16.904412 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.904378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerStarted","Data":"a003a8e0e0556ebd1f4485414deb965bed916016b7d55548433ac93dd5a07ba0"}
Jul 17 22:20:16.904630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.904431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerStarted","Data":"52d0c5fa432bfc061b266d415c4fa2ebab892ba4a3f0bc89d8b96686ef81b3b0"}
Jul 17 22:20:16.924730 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:16.924632    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" podStartSLOduration=0.924592068 podStartE2EDuration="924.592068ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:20:16.923523132 +0000 UTC m=+3180.835763421" watchObservedRunningTime="2026-07-17 22:20:16.924592068 +0000 UTC m=+3180.836832358"
Jul 17 22:20:17.173161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.168503    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"]
Jul 17 22:20:17.173161 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.171937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.179125 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.179091    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"kube-root-ca.crt\""
Jul 17 22:20:17.179981 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.179955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"default-dockercfg-99xqt\""
Jul 17 22:20:17.180094 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.179955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"openshift-service-ca.crt\""
Jul 17 22:20:17.180094 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.180006    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"sub-test-model-5401e5-kserve-self-signed-certs\""
Jul 17 22:20:17.197041 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.197012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"]
Jul 17 22:20:17.260028 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.260002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tmp-dir\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.260148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.260061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tls-certs\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.260148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.260091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7w9g\" (UniqueName: \"kubernetes.io/projected/49368892-14e6-43f6-b4e6-7a1f9dcd9923-kube-api-access-x7w9g\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.260263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.260164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-model-cache\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.260263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.260227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-home\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.260263 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.260259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-dshm\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361015 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.360975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-home\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361015 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-dshm\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361288 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tmp-dir\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361288 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tls-certs\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361288 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7w9g\" (UniqueName: \"kubernetes.io/projected/49368892-14e6-43f6-b4e6-7a1f9dcd9923-kube-api-access-x7w9g\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361288 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-model-cache\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-home\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tmp-dir\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.361580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.361564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-model-cache\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.363568 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.363546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-dshm\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.363845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.363828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tls-certs\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.370350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.370329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7w9g\" (UniqueName: \"kubernetes.io/projected/49368892-14e6-43f6-b4e6-7a1f9dcd9923-kube-api-access-x7w9g\") pod \"sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") " pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.482943 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.482850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:17.635379 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.635355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"]
Jul 17 22:20:17.637560 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:20:17.637531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49368892_14e6_43f6_b4e6_7a1f9dcd9923.slice/crio-32564b94f4d823d0f59f0c660e9090a8af9b231a12bee2a9bea631e190db19f2 WatchSource:0}: Error finding container 32564b94f4d823d0f59f0c660e9090a8af9b231a12bee2a9bea631e190db19f2: Status 404 returned error can't find the container with id 32564b94f4d823d0f59f0c660e9090a8af9b231a12bee2a9bea631e190db19f2
Jul 17 22:20:17.911198 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.911101    2563 generic.go:356] "Generic (PLEG): container finished" podID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerID="a003a8e0e0556ebd1f4485414deb965bed916016b7d55548433ac93dd5a07ba0" exitCode=6
Jul 17 22:20:17.911366 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.911199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerDied","Data":"a003a8e0e0556ebd1f4485414deb965bed916016b7d55548433ac93dd5a07ba0"}
Jul 17 22:20:17.911366 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.911349    2563 scope.go:117] "RemoveContainer" containerID="a003a8e0e0556ebd1f4485414deb965bed916016b7d55548433ac93dd5a07ba0"
Jul 17 22:20:17.912892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.912871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm" event={"ID":"49368892-14e6-43f6-b4e6-7a1f9dcd9923","Type":"ContainerStarted","Data":"651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612"}
Jul 17 22:20:17.912892 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.912899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm" event={"ID":"49368892-14e6-43f6-b4e6-7a1f9dcd9923","Type":"ContainerStarted","Data":"32564b94f4d823d0f59f0c660e9090a8af9b231a12bee2a9bea631e190db19f2"}
Jul 17 22:20:17.978304 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:17.978254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm" podStartSLOduration=0.978238032 podStartE2EDuration="978.238032ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:20:17.967894953 +0000 UTC m=+3181.880135241" watchObservedRunningTime="2026-07-17 22:20:17.978238032 +0000 UTC m=+3181.890478320"
Jul 17 22:20:18.919374 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:18.919335    2563 generic.go:356] "Generic (PLEG): container finished" podID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerID="0128e142c10278b130090da294ed4f5eba2ca845b82be17f4235d6d557b17601" exitCode=6
Jul 17 22:20:18.919894 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:18.919418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerDied","Data":"0128e142c10278b130090da294ed4f5eba2ca845b82be17f4235d6d557b17601"}
Jul 17 22:20:18.919894 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:18.919479    2563 scope.go:117] "RemoveContainer" containerID="a003a8e0e0556ebd1f4485414deb965bed916016b7d55548433ac93dd5a07ba0"
Jul 17 22:20:18.919894 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:18.919674    2563 scope.go:117] "RemoveContainer" containerID="0128e142c10278b130090da294ed4f5eba2ca845b82be17f4235d6d557b17601"
Jul 17 22:20:18.920033 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:18.919914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4_odh-ai-gateway-infra(86b12311-e11e-4640-a6e2-07ed6b88785a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a"
Jul 17 22:20:19.926421 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:19.926394    2563 scope.go:117] "RemoveContainer" containerID="0128e142c10278b130090da294ed4f5eba2ca845b82be17f4235d6d557b17601"
Jul 17 22:20:19.926900 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:19.926576    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4_odh-ai-gateway-infra(86b12311-e11e-4640-a6e2-07ed6b88785a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a"
Jul 17 22:20:25.597154 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:25.597117    2563 scope.go:117] "RemoveContainer" containerID="6620faa8b69d42a06a45d28e73fdafd8b38d9bbf341a7ce5e34282ff52540a2d"
Jul 17 22:20:25.960258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:25.960219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" event={"ID":"3587e7da-f94e-4488-8780-7d5f50148336","Type":"ContainerStarted","Data":"c79ee2ddfc80e68fc740c972e335534ae8cba4b31c493b9befacb0ae5fc7d1d3"}
Jul 17 22:20:25.992745 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:25.992693    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" podStartSLOduration=17.992678485 podStartE2EDuration="17.992678485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:20:25.991039791 +0000 UTC m=+3189.903280081" watchObservedRunningTime="2026-07-17 22:20:25.992678485 +0000 UTC m=+3189.904918786"
Jul 17 22:20:26.967437 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:26.967401    2563 generic.go:356] "Generic (PLEG): container finished" podID="3587e7da-f94e-4488-8780-7d5f50148336" containerID="c79ee2ddfc80e68fc740c972e335534ae8cba4b31c493b9befacb0ae5fc7d1d3" exitCode=6
Jul 17 22:20:26.967936 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:26.967468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" event={"ID":"3587e7da-f94e-4488-8780-7d5f50148336","Type":"ContainerDied","Data":"c79ee2ddfc80e68fc740c972e335534ae8cba4b31c493b9befacb0ae5fc7d1d3"}
Jul 17 22:20:26.967936 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:26.967529    2563 scope.go:117] "RemoveContainer" containerID="6620faa8b69d42a06a45d28e73fdafd8b38d9bbf341a7ce5e34282ff52540a2d"
Jul 17 22:20:26.967936 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:26.967819    2563 scope.go:117] "RemoveContainer" containerID="c79ee2ddfc80e68fc740c972e335534ae8cba4b31c493b9befacb0ae5fc7d1d3"
Jul 17 22:20:26.968114 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:26.968024    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj_odh-ai-gateway-infra(3587e7da-f94e-4488-8780-7d5f50148336)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" podUID="3587e7da-f94e-4488-8780-7d5f50148336"
Jul 17 22:20:27.002789 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:27.002752    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"]
Jul 17 22:20:27.483002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:27.482968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:27.483184 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:27.483036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:27.496077 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:27.496049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:27.989320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:27.989292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:20:28.132711 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.132690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:28.263952 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.263860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587e7da-f94e-4488-8780-7d5f50148336-maas-api-service-ca\") pod \"3587e7da-f94e-4488-8780-7d5f50148336\" (UID: \"3587e7da-f94e-4488-8780-7d5f50148336\") "
Jul 17 22:20:28.264301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.264270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3587e7da-f94e-4488-8780-7d5f50148336-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3587e7da-f94e-4488-8780-7d5f50148336" (UID: "3587e7da-f94e-4488-8780-7d5f50148336"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:28.365109 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.365074    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587e7da-f94e-4488-8780-7d5f50148336-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:28.981419 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.981386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"
Jul 17 22:20:28.981642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.981426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj" event={"ID":"3587e7da-f94e-4488-8780-7d5f50148336","Type":"ContainerDied","Data":"9802513b8a8257828edbea4bf1ab0c0dc9177b0ab09983a922fae2a43f08bcb6"}
Jul 17 22:20:28.981642 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.981482    2563 scope.go:117] "RemoveContainer" containerID="c79ee2ddfc80e68fc740c972e335534ae8cba4b31c493b9befacb0ae5fc7d1d3"
Jul 17 22:20:28.997308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:28.997283    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"]
Jul 17 22:20:29.001407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:29.001386    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-d5qgj"]
Jul 17 22:20:30.608573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:30.608537    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3587e7da-f94e-4488-8780-7d5f50148336" path="/var/lib/kubelet/pods/3587e7da-f94e-4488-8780-7d5f50148336/volumes"
Jul 17 22:20:34.597176 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:34.597142    2563 scope.go:117] "RemoveContainer" containerID="0128e142c10278b130090da294ed4f5eba2ca845b82be17f4235d6d557b17601"
Jul 17 22:20:35.014190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:35.014155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerStarted","Data":"dec897adb60199ad63b270e43271a97104c0823f396c609e85442b794161a9a0"}
Jul 17 22:20:36.020726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:36.020693    2563 generic.go:356] "Generic (PLEG): container finished" podID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerID="dec897adb60199ad63b270e43271a97104c0823f396c609e85442b794161a9a0" exitCode=6
Jul 17 22:20:36.021163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:36.020770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerDied","Data":"dec897adb60199ad63b270e43271a97104c0823f396c609e85442b794161a9a0"}
Jul 17 22:20:36.021163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:36.020846    2563 scope.go:117] "RemoveContainer" containerID="0128e142c10278b130090da294ed4f5eba2ca845b82be17f4235d6d557b17601"
Jul 17 22:20:36.021163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:36.021132    2563 scope.go:117] "RemoveContainer" containerID="dec897adb60199ad63b270e43271a97104c0823f396c609e85442b794161a9a0"
Jul 17 22:20:36.021395 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:36.021375    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4_odh-ai-gateway-infra(86b12311-e11e-4640-a6e2-07ed6b88785a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a"
Jul 17 22:20:36.050079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:36.050046    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"]
Jul 17 22:20:37.180256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:37.180220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:37.350777 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:37.350676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b12311-e11e-4640-a6e2-07ed6b88785a-maas-api-service-ca\") pod \"86b12311-e11e-4640-a6e2-07ed6b88785a\" (UID: \"86b12311-e11e-4640-a6e2-07ed6b88785a\") "
Jul 17 22:20:37.351042 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:37.351016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b12311-e11e-4640-a6e2-07ed6b88785a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86b12311-e11e-4640-a6e2-07ed6b88785a" (UID: "86b12311-e11e-4640-a6e2-07ed6b88785a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:37.452027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:37.451980    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b12311-e11e-4640-a6e2-07ed6b88785a-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:38.039981 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:38.039949    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"
Jul 17 22:20:38.040160 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:38.039948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4" event={"ID":"86b12311-e11e-4640-a6e2-07ed6b88785a","Type":"ContainerDied","Data":"52d0c5fa432bfc061b266d415c4fa2ebab892ba4a3f0bc89d8b96686ef81b3b0"}
Jul 17 22:20:38.040160 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:38.040093    2563 scope.go:117] "RemoveContainer" containerID="dec897adb60199ad63b270e43271a97104c0823f396c609e85442b794161a9a0"
Jul 17 22:20:38.063416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:38.063380    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"]
Jul 17 22:20:38.067357 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:38.067331    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wd2w4"]
Jul 17 22:20:38.602553 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:38.602519    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" path="/var/lib/kubelet/pods/86b12311-e11e-4640-a6e2-07ed6b88785a/volumes"
Jul 17 22:20:40.457178 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457142    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"]
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457534    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457539    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457545    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457551    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457562 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457562    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457572    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457585    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457702    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457710    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457716    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457723    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.457837 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.457729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3587e7da-f94e-4488-8780-7d5f50148336" containerName="revoke-keys"
Jul 17 22:20:40.462239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.462221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:40.464438 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.464420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:20:40.467784 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.467758    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"]
Jul 17 22:20:40.581209 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.581169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d315e74-4967-4643-88e9-375712254417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4\" (UID: \"2d315e74-4967-4643-88e9-375712254417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:40.682364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.682333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d315e74-4967-4643-88e9-375712254417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4\" (UID: \"2d315e74-4967-4643-88e9-375712254417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:40.683003 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.682984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d315e74-4967-4643-88e9-375712254417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4\" (UID: \"2d315e74-4967-4643-88e9-375712254417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:40.775581 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.775489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:40.912353 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:40.912283    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"]
Jul 17 22:20:40.914627 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:20:40.914587    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d315e74_4967_4643_88e9_375712254417.slice/crio-836fb87d8001688f70fec00a8913d7dfa2d8badc8a694b055f7ebe3c041610fd WatchSource:0}: Error finding container 836fb87d8001688f70fec00a8913d7dfa2d8badc8a694b055f7ebe3c041610fd: Status 404 returned error can't find the container with id 836fb87d8001688f70fec00a8913d7dfa2d8badc8a694b055f7ebe3c041610fd
Jul 17 22:20:41.058061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:41.057992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" event={"ID":"2d315e74-4967-4643-88e9-375712254417","Type":"ContainerStarted","Data":"836fb87d8001688f70fec00a8913d7dfa2d8badc8a694b055f7ebe3c041610fd"}
Jul 17 22:20:42.064767 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:42.064727    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d315e74-4967-4643-88e9-375712254417" containerID="681c4e4631b4458d188b9e72e92c7edf89b4496df7cc4dea1f63dd1bad8ee1bc" exitCode=6
Jul 17 22:20:42.065135 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:42.064783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" event={"ID":"2d315e74-4967-4643-88e9-375712254417","Type":"ContainerDied","Data":"681c4e4631b4458d188b9e72e92c7edf89b4496df7cc4dea1f63dd1bad8ee1bc"}
Jul 17 22:20:42.065135 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:42.065046    2563 scope.go:117] "RemoveContainer" containerID="681c4e4631b4458d188b9e72e92c7edf89b4496df7cc4dea1f63dd1bad8ee1bc"
Jul 17 22:20:43.071738 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.071698    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d315e74-4967-4643-88e9-375712254417" containerID="15ab2f864be630fb8533d33fbc5459791977a12f5b85a3d6fec8c14907ae6fc2" exitCode=6
Jul 17 22:20:43.072234 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.071781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" event={"ID":"2d315e74-4967-4643-88e9-375712254417","Type":"ContainerDied","Data":"15ab2f864be630fb8533d33fbc5459791977a12f5b85a3d6fec8c14907ae6fc2"}
Jul 17 22:20:43.072234 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.071853    2563 scope.go:117] "RemoveContainer" containerID="681c4e4631b4458d188b9e72e92c7edf89b4496df7cc4dea1f63dd1bad8ee1bc"
Jul 17 22:20:43.072234 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.072059    2563 scope.go:117] "RemoveContainer" containerID="15ab2f864be630fb8533d33fbc5459791977a12f5b85a3d6fec8c14907ae6fc2"
Jul 17 22:20:43.072369 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:43.072284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4_odh-ai-gateway-infra(2d315e74-4967-4643-88e9-375712254417)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" podUID="2d315e74-4967-4643-88e9-375712254417"
Jul 17 22:20:43.879035 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.879006    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"]
Jul 17 22:20:43.879446 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.879434    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b12311-e11e-4640-a6e2-07ed6b88785a" containerName="revoke-keys"
Jul 17 22:20:43.883736 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.883720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:20:43.890436 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.890408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"]
Jul 17 22:20:43.912084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:43.912052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e418757-09e1-44d7-bb4c-5b5effb16ab5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z\" (UID: \"8e418757-09e1-44d7-bb4c-5b5effb16ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:20:44.013070 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:44.013034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e418757-09e1-44d7-bb4c-5b5effb16ab5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z\" (UID: \"8e418757-09e1-44d7-bb4c-5b5effb16ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:20:44.013693 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:44.013677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e418757-09e1-44d7-bb4c-5b5effb16ab5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z\" (UID: \"8e418757-09e1-44d7-bb4c-5b5effb16ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:20:44.078813 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:44.078786    2563 scope.go:117] "RemoveContainer" containerID="15ab2f864be630fb8533d33fbc5459791977a12f5b85a3d6fec8c14907ae6fc2"
Jul 17 22:20:44.079170 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:44.078977    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4_odh-ai-gateway-infra(2d315e74-4967-4643-88e9-375712254417)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" podUID="2d315e74-4967-4643-88e9-375712254417"
Jul 17 22:20:44.196250 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:44.196212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:20:44.337781 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:44.337754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"]
Jul 17 22:20:44.339019 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:20:44.338989    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e418757_09e1_44d7_bb4c_5b5effb16ab5.slice/crio-901c4f87f81b32698f2ea3af0eca199b80a226c6acf57706d3e0a4964958da9d WatchSource:0}: Error finding container 901c4f87f81b32698f2ea3af0eca199b80a226c6acf57706d3e0a4964958da9d: Status 404 returned error can't find the container with id 901c4f87f81b32698f2ea3af0eca199b80a226c6acf57706d3e0a4964958da9d
Jul 17 22:20:45.085416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:45.085381    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerID="df5a1daef5d4a9ea3e2d5da142b93f55cbda1a5eec72f6bd70fd4a2e209819f7" exitCode=6
Jul 17 22:20:45.085760 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:45.085475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" event={"ID":"8e418757-09e1-44d7-bb4c-5b5effb16ab5","Type":"ContainerDied","Data":"df5a1daef5d4a9ea3e2d5da142b93f55cbda1a5eec72f6bd70fd4a2e209819f7"}
Jul 17 22:20:45.085760 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:45.085538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" event={"ID":"8e418757-09e1-44d7-bb4c-5b5effb16ab5","Type":"ContainerStarted","Data":"901c4f87f81b32698f2ea3af0eca199b80a226c6acf57706d3e0a4964958da9d"}
Jul 17 22:20:45.085760 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:45.085675    2563 scope.go:117] "RemoveContainer" containerID="df5a1daef5d4a9ea3e2d5da142b93f55cbda1a5eec72f6bd70fd4a2e209819f7"
Jul 17 22:20:46.093014 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:46.092921    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerID="68c72eb821abe0a22dab6a869d35e0071076e15353234bbc53827330323b36e9" exitCode=6
Jul 17 22:20:46.093450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:46.093007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" event={"ID":"8e418757-09e1-44d7-bb4c-5b5effb16ab5","Type":"ContainerDied","Data":"68c72eb821abe0a22dab6a869d35e0071076e15353234bbc53827330323b36e9"}
Jul 17 22:20:46.093450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:46.093083    2563 scope.go:117] "RemoveContainer" containerID="df5a1daef5d4a9ea3e2d5da142b93f55cbda1a5eec72f6bd70fd4a2e209819f7"
Jul 17 22:20:46.093450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:46.093320    2563 scope.go:117] "RemoveContainer" containerID="68c72eb821abe0a22dab6a869d35e0071076e15353234bbc53827330323b36e9"
Jul 17 22:20:46.093572 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:46.093539    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z_odh-ai-gateway-infra(8e418757-09e1-44d7-bb4c-5b5effb16ab5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5"
Jul 17 22:20:47.100303 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:47.100275    2563 scope.go:117] "RemoveContainer" containerID="68c72eb821abe0a22dab6a869d35e0071076e15353234bbc53827330323b36e9"
Jul 17 22:20:47.100714 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:47.100465    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z_odh-ai-gateway-infra(8e418757-09e1-44d7-bb4c-5b5effb16ab5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5"
Jul 17 22:20:54.597381 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:54.597346    2563 scope.go:117] "RemoveContainer" containerID="15ab2f864be630fb8533d33fbc5459791977a12f5b85a3d6fec8c14907ae6fc2"
Jul 17 22:20:55.144409 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:55.144327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" event={"ID":"2d315e74-4967-4643-88e9-375712254417","Type":"ContainerStarted","Data":"5afe0ad89ee6829dd55a7477a9dcc0a71fc7f9c550d237d8cb5bdcfd4b76364e"}
Jul 17 22:20:55.159785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:55.159737    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" podStartSLOduration=15.159724102 podStartE2EDuration="15.159724102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:20:55.158905137 +0000 UTC m=+3219.071145428" watchObservedRunningTime="2026-07-17 22:20:55.159724102 +0000 UTC m=+3219.071964389"
Jul 17 22:20:56.154385 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:56.154350    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d315e74-4967-4643-88e9-375712254417" containerID="5afe0ad89ee6829dd55a7477a9dcc0a71fc7f9c550d237d8cb5bdcfd4b76364e" exitCode=6
Jul 17 22:20:56.154890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:56.154421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" event={"ID":"2d315e74-4967-4643-88e9-375712254417","Type":"ContainerDied","Data":"5afe0ad89ee6829dd55a7477a9dcc0a71fc7f9c550d237d8cb5bdcfd4b76364e"}
Jul 17 22:20:56.154890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:56.154489    2563 scope.go:117] "RemoveContainer" containerID="15ab2f864be630fb8533d33fbc5459791977a12f5b85a3d6fec8c14907ae6fc2"
Jul 17 22:20:56.154890 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:56.154842    2563 scope.go:117] "RemoveContainer" containerID="5afe0ad89ee6829dd55a7477a9dcc0a71fc7f9c550d237d8cb5bdcfd4b76364e"
Jul 17 22:20:56.155158 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:20:56.155124    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4_odh-ai-gateway-infra(2d315e74-4967-4643-88e9-375712254417)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" podUID="2d315e74-4967-4643-88e9-375712254417"
Jul 17 22:20:56.183967 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:56.183940    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"]
Jul 17 22:20:57.299817 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:57.299791    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:57.331435 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:57.331405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d315e74-4967-4643-88e9-375712254417-maas-api-service-ca\") pod \"2d315e74-4967-4643-88e9-375712254417\" (UID: \"2d315e74-4967-4643-88e9-375712254417\") "
Jul 17 22:20:57.331875 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:57.331847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d315e74-4967-4643-88e9-375712254417-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d315e74-4967-4643-88e9-375712254417" (UID: "2d315e74-4967-4643-88e9-375712254417"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:57.432808 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:57.432730    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d315e74-4967-4643-88e9-375712254417-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:20:58.167982 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:58.167948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4" event={"ID":"2d315e74-4967-4643-88e9-375712254417","Type":"ContainerDied","Data":"836fb87d8001688f70fec00a8913d7dfa2d8badc8a694b055f7ebe3c041610fd"}
Jul 17 22:20:58.168177 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:58.168015    2563 scope.go:117] "RemoveContainer" containerID="5afe0ad89ee6829dd55a7477a9dcc0a71fc7f9c550d237d8cb5bdcfd4b76364e"
Jul 17 22:20:58.168177 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:58.167967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"
Jul 17 22:20:58.189048 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:58.189021    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"]
Jul 17 22:20:58.193370 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:58.193344    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-g6bl4"]
Jul 17 22:20:58.603332 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:58.603252    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d315e74-4967-4643-88e9-375712254417" path="/var/lib/kubelet/pods/2d315e74-4967-4643-88e9-375712254417/volumes"
Jul 17 22:20:59.596989 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:20:59.596957    2563 scope.go:117] "RemoveContainer" containerID="68c72eb821abe0a22dab6a869d35e0071076e15353234bbc53827330323b36e9"
Jul 17 22:21:00.181172 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:00.181133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" event={"ID":"8e418757-09e1-44d7-bb4c-5b5effb16ab5","Type":"ContainerStarted","Data":"92c88cc1a96ef14cc204d1495b12e68be42bea841908fd42a19692375ece5c66"}
Jul 17 22:21:00.197959 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:00.197907    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" podStartSLOduration=17.19789256 podStartE2EDuration="17.19789256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:21:00.195810796 +0000 UTC m=+3224.108051126" watchObservedRunningTime="2026-07-17 22:21:00.19789256 +0000 UTC m=+3224.110132848"
Jul 17 22:21:01.187985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:01.187952    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerID="92c88cc1a96ef14cc204d1495b12e68be42bea841908fd42a19692375ece5c66" exitCode=6
Jul 17 22:21:01.188441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:01.188001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" event={"ID":"8e418757-09e1-44d7-bb4c-5b5effb16ab5","Type":"ContainerDied","Data":"92c88cc1a96ef14cc204d1495b12e68be42bea841908fd42a19692375ece5c66"}
Jul 17 22:21:01.188441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:01.188075    2563 scope.go:117] "RemoveContainer" containerID="68c72eb821abe0a22dab6a869d35e0071076e15353234bbc53827330323b36e9"
Jul 17 22:21:01.188530 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:01.188439    2563 scope.go:117] "RemoveContainer" containerID="92c88cc1a96ef14cc204d1495b12e68be42bea841908fd42a19692375ece5c66"
Jul 17 22:21:01.188680 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:01.188662    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z_odh-ai-gateway-infra(8e418757-09e1-44d7-bb4c-5b5effb16ab5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5"
Jul 17 22:21:01.217740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:01.211729    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"]
Jul 17 22:21:02.335671 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:02.335648    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:21:02.377516 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:02.377488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e418757-09e1-44d7-bb4c-5b5effb16ab5-maas-api-service-ca\") pod \"8e418757-09e1-44d7-bb4c-5b5effb16ab5\" (UID: \"8e418757-09e1-44d7-bb4c-5b5effb16ab5\") "
Jul 17 22:21:02.377920 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:02.377899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e418757-09e1-44d7-bb4c-5b5effb16ab5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e418757-09e1-44d7-bb4c-5b5effb16ab5" (UID: "8e418757-09e1-44d7-bb4c-5b5effb16ab5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:02.478945 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:02.478862    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e418757-09e1-44d7-bb4c-5b5effb16ab5-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:03.201583 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:03.201550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"
Jul 17 22:21:03.201583 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:03.201562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z" event={"ID":"8e418757-09e1-44d7-bb4c-5b5effb16ab5","Type":"ContainerDied","Data":"901c4f87f81b32698f2ea3af0eca199b80a226c6acf57706d3e0a4964958da9d"}
Jul 17 22:21:03.201861 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:03.201637    2563 scope.go:117] "RemoveContainer" containerID="92c88cc1a96ef14cc204d1495b12e68be42bea841908fd42a19692375ece5c66"
Jul 17 22:21:03.217977 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:03.217949    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"]
Jul 17 22:21:03.223582 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:03.223559    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-8gz2z"]
Jul 17 22:21:03.707994 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:03.707959    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"]
Jul 17 22:21:04.602839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:04.602805    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" path="/var/lib/kubelet/pods/8e418757-09e1-44d7-bb4c-5b5effb16ab5/volumes"
Jul 17 22:21:04.779904 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:04.779870    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"]
Jul 17 22:21:08.272673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.272580    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"]
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.272965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.272976    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.272988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.272993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.272999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273005    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273012    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273017    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273028    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273092    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.273350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273101    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273108    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273113    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:08.273350 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.273119    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:08.277558 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.277541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:08.280119 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.280090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:21:08.282636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.282590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"]
Jul 17 22:21:08.331194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.331161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8\" (UID: \"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:08.431914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.431883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8\" (UID: \"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:08.432550 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.432529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8\" (UID: \"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:08.590552 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.590461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:08.726527 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:08.726501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"]
Jul 17 22:21:08.728855 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:21:08.728829    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce4bcbb_443a_429e_96be_a9b6ecbe81ef.slice/crio-2469e4103c59b03519cbd46e7982d73da84055ed72d8f2c0007935cc47218427 WatchSource:0}: Error finding container 2469e4103c59b03519cbd46e7982d73da84055ed72d8f2c0007935cc47218427: Status 404 returned error can't find the container with id 2469e4103c59b03519cbd46e7982d73da84055ed72d8f2c0007935cc47218427
Jul 17 22:21:09.235993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:09.235897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" event={"ID":"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef","Type":"ContainerStarted","Data":"216cc383cfcc9434bca8512591ca341c856bdb96b6836a5a3f127d95bcbc5d39"}
Jul 17 22:21:09.235993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:09.235955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" event={"ID":"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef","Type":"ContainerStarted","Data":"2469e4103c59b03519cbd46e7982d73da84055ed72d8f2c0007935cc47218427"}
Jul 17 22:21:09.251489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:09.251446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" podStartSLOduration=1.251431682 podStartE2EDuration="1.251431682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:21:09.249901202 +0000 UTC m=+3233.162141492" watchObservedRunningTime="2026-07-17 22:21:09.251431682 +0000 UTC m=+3233.163671970"
Jul 17 22:21:10.242494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:10.242453    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerID="216cc383cfcc9434bca8512591ca341c856bdb96b6836a5a3f127d95bcbc5d39" exitCode=6
Jul 17 22:21:10.242974 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:10.242508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" event={"ID":"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef","Type":"ContainerDied","Data":"216cc383cfcc9434bca8512591ca341c856bdb96b6836a5a3f127d95bcbc5d39"}
Jul 17 22:21:10.242974 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:10.242750    2563 scope.go:117] "RemoveContainer" containerID="216cc383cfcc9434bca8512591ca341c856bdb96b6836a5a3f127d95bcbc5d39"
Jul 17 22:21:11.249354 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:11.249319    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerID="eb547748d78fef3c6481c45625202b95cd4faac13ba13f6ae3942e6591181767" exitCode=6
Jul 17 22:21:11.249824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:11.249408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" event={"ID":"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef","Type":"ContainerDied","Data":"eb547748d78fef3c6481c45625202b95cd4faac13ba13f6ae3942e6591181767"}
Jul 17 22:21:11.249824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:11.249480    2563 scope.go:117] "RemoveContainer" containerID="216cc383cfcc9434bca8512591ca341c856bdb96b6836a5a3f127d95bcbc5d39"
Jul 17 22:21:11.249824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:11.249576    2563 scope.go:117] "RemoveContainer" containerID="eb547748d78fef3c6481c45625202b95cd4faac13ba13f6ae3942e6591181767"
Jul 17 22:21:11.249824 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:11.249810    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8_odh-ai-gateway-infra(0ce4bcbb-443a-429e-96be-a9b6ecbe81ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef"
Jul 17 22:21:12.258088 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:12.258059    2563 scope.go:117] "RemoveContainer" containerID="eb547748d78fef3c6481c45625202b95cd4faac13ba13f6ae3942e6591181767"
Jul 17 22:21:12.258473 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:12.258296    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8_odh-ai-gateway-infra(0ce4bcbb-443a-429e-96be-a9b6ecbe81ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef"
Jul 17 22:21:16.186927 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.186896    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"]
Jul 17 22:21:16.187399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.187282    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:16.187399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.187293    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d315e74-4967-4643-88e9-375712254417" containerName="revoke-keys"
Jul 17 22:21:16.187399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.187351    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e418757-09e1-44d7-bb4c-5b5effb16ab5" containerName="revoke-keys"
Jul 17 22:21:16.192102 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.192077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:16.200970 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.200934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"]
Jul 17 22:21:16.303845 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.303810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe33f76-6610-46e3-990c-b678fd2052f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm\" (UID: \"fbe33f76-6610-46e3-990c-b678fd2052f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:16.404663 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.404623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe33f76-6610-46e3-990c-b678fd2052f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm\" (UID: \"fbe33f76-6610-46e3-990c-b678fd2052f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:16.405209 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.405189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe33f76-6610-46e3-990c-b678fd2052f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm\" (UID: \"fbe33f76-6610-46e3-990c-b678fd2052f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:16.504138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.504055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:16.657434 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:16.657409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"]
Jul 17 22:21:16.659199 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:21:16.659171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe33f76_6610_46e3_990c_b678fd2052f0.slice/crio-7a924b82ca645ef950480e582be4c2cc9856e16283e8c10b6570175512175a3c WatchSource:0}: Error finding container 7a924b82ca645ef950480e582be4c2cc9856e16283e8c10b6570175512175a3c: Status 404 returned error can't find the container with id 7a924b82ca645ef950480e582be4c2cc9856e16283e8c10b6570175512175a3c
Jul 17 22:21:17.260576 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:17.260544    2563 scope.go:117] "RemoveContainer" containerID="1a758e38f6806476f693616783ff9a04980ca54a54a9a8be5c074ee5ff3bc3b4"
Jul 17 22:21:17.270145 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:17.270118    2563 scope.go:117] "RemoveContainer" containerID="49967041d6b5465397916abdb2dcf1502c29be3d378001916bb8fcfdc6cd8f29"
Jul 17 22:21:17.285079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:17.285050    2563 generic.go:356] "Generic (PLEG): container finished" podID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerID="590fac881fccb6147eccca8e1e2b52e4f213e1f346c7c15bd50365b451429334" exitCode=6
Jul 17 22:21:17.285225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:17.285136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" event={"ID":"fbe33f76-6610-46e3-990c-b678fd2052f0","Type":"ContainerDied","Data":"590fac881fccb6147eccca8e1e2b52e4f213e1f346c7c15bd50365b451429334"}
Jul 17 22:21:17.285225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:17.285184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" event={"ID":"fbe33f76-6610-46e3-990c-b678fd2052f0","Type":"ContainerStarted","Data":"7a924b82ca645ef950480e582be4c2cc9856e16283e8c10b6570175512175a3c"}
Jul 17 22:21:17.285348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:17.285326    2563 scope.go:117] "RemoveContainer" containerID="590fac881fccb6147eccca8e1e2b52e4f213e1f346c7c15bd50365b451429334"
Jul 17 22:21:18.293916 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:18.293879    2563 generic.go:356] "Generic (PLEG): container finished" podID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerID="574e58a7128d9e20153ede471c594bd389f788f042285bc283158312112d292d" exitCode=6
Jul 17 22:21:18.294370 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:18.293964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" event={"ID":"fbe33f76-6610-46e3-990c-b678fd2052f0","Type":"ContainerDied","Data":"574e58a7128d9e20153ede471c594bd389f788f042285bc283158312112d292d"}
Jul 17 22:21:18.294370 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:18.294036    2563 scope.go:117] "RemoveContainer" containerID="590fac881fccb6147eccca8e1e2b52e4f213e1f346c7c15bd50365b451429334"
Jul 17 22:21:18.294370 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:18.294237    2563 scope.go:117] "RemoveContainer" containerID="574e58a7128d9e20153ede471c594bd389f788f042285bc283158312112d292d"
Jul 17 22:21:18.294499 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:18.294471    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm_odh-ai-gateway-infra(fbe33f76-6610-46e3-990c-b678fd2052f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0"
Jul 17 22:21:18.728689 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:18.728639    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5" podUID="5ae8b611-14d9-48ff-8651-4358f85d03cf" containerName="main" containerID="cri-o://59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827" gracePeriod=45
Jul 17 22:21:18.982030 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:18.981966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:21:19.130229 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae8b611-14d9-48ff-8651-4358f85d03cf-tls-certs\") pod \"5ae8b611-14d9-48ff-8651-4358f85d03cf\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") "
Jul 17 22:21:19.130229 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-home\") pod \"5ae8b611-14d9-48ff-8651-4358f85d03cf\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") "
Jul 17 22:21:19.130495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-model-cache\") pod \"5ae8b611-14d9-48ff-8651-4358f85d03cf\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") "
Jul 17 22:21:19.130495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-tmp-dir\") pod \"5ae8b611-14d9-48ff-8651-4358f85d03cf\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") "
Jul 17 22:21:19.130495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130309    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gdlx\" (UniqueName: \"kubernetes.io/projected/5ae8b611-14d9-48ff-8651-4358f85d03cf-kube-api-access-9gdlx\") pod \"5ae8b611-14d9-48ff-8651-4358f85d03cf\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") "
Jul 17 22:21:19.130495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-dshm\") pod \"5ae8b611-14d9-48ff-8651-4358f85d03cf\" (UID: \"5ae8b611-14d9-48ff-8651-4358f85d03cf\") "
Jul 17 22:21:19.130788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-home" (OuterVolumeSpecName: "home") pod "5ae8b611-14d9-48ff-8651-4358f85d03cf" (UID: "5ae8b611-14d9-48ff-8651-4358f85d03cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:19.130788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130566    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ae8b611-14d9-48ff-8651-4358f85d03cf" (UID: "5ae8b611-14d9-48ff-8651-4358f85d03cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:19.130788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ae8b611-14d9-48ff-8651-4358f85d03cf" (UID: "5ae8b611-14d9-48ff-8651-4358f85d03cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:19.130788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130716    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:19.130788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130738    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:19.130788 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.130757    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:19.132630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.132581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-dshm" (OuterVolumeSpecName: "dshm") pod "5ae8b611-14d9-48ff-8651-4358f85d03cf" (UID: "5ae8b611-14d9-48ff-8651-4358f85d03cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:19.132630 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.132590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae8b611-14d9-48ff-8651-4358f85d03cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ae8b611-14d9-48ff-8651-4358f85d03cf" (UID: "5ae8b611-14d9-48ff-8651-4358f85d03cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:21:19.132765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.132685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae8b611-14d9-48ff-8651-4358f85d03cf-kube-api-access-9gdlx" (OuterVolumeSpecName: "kube-api-access-9gdlx") pod "5ae8b611-14d9-48ff-8651-4358f85d03cf" (UID: "5ae8b611-14d9-48ff-8651-4358f85d03cf"). InnerVolumeSpecName "kube-api-access-9gdlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:21:19.231710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.231668    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae8b611-14d9-48ff-8651-4358f85d03cf-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:19.231710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.231702    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gdlx\" (UniqueName: \"kubernetes.io/projected/5ae8b611-14d9-48ff-8651-4358f85d03cf-kube-api-access-9gdlx\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:19.231710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.231711    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae8b611-14d9-48ff-8651-4358f85d03cf-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:19.300468 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.300382    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ae8b611-14d9-48ff-8651-4358f85d03cf" containerID="59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827" exitCode=0
Jul 17 22:21:19.300468 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.300431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"
Jul 17 22:21:19.300993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.300465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5" event={"ID":"5ae8b611-14d9-48ff-8651-4358f85d03cf","Type":"ContainerDied","Data":"59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827"}
Jul 17 22:21:19.300993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.300515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5" event={"ID":"5ae8b611-14d9-48ff-8651-4358f85d03cf","Type":"ContainerDied","Data":"8bcc2cda1c656d74e6582a0ff00744345e01734785889d960c8a0b8a8f1ef7e8"}
Jul 17 22:21:19.300993 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.300540    2563 scope.go:117] "RemoveContainer" containerID="59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827"
Jul 17 22:21:19.302407 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.302389    2563 scope.go:117] "RemoveContainer" containerID="574e58a7128d9e20153ede471c594bd389f788f042285bc283158312112d292d"
Jul 17 22:21:19.302610 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:19.302574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm_odh-ai-gateway-infra(fbe33f76-6610-46e3-990c-b678fd2052f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0"
Jul 17 22:21:19.310187 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.310171    2563 scope.go:117] "RemoveContainer" containerID="59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827"
Jul 17 22:21:19.310430 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:19.310411    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827\": container with ID starting with 59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827 not found: ID does not exist" containerID="59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827"
Jul 17 22:21:19.310500 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.310444    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827"} err="failed to get container status \"59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827\": rpc error: code = NotFound desc = could not find container \"59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827\": container with ID starting with 59c0bfb0560cbae09d8cf6968e14d68a46a2d9ed039202e21eb6178cee113827 not found: ID does not exist"
Jul 17 22:21:19.329398 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.329360    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"]
Jul 17 22:21:19.332839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.332816    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/sub-test-model-8e6a3a-kserve-766c98cd59-lvsw5"]
Jul 17 22:21:19.800765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:19.800728    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm" podUID="49368892-14e6-43f6-b4e6-7a1f9dcd9923" containerName="main" containerID="cri-o://651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612" gracePeriod=45
Jul 17 22:21:20.068591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.068567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:21:20.138573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-home\") pod \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") "
Jul 17 22:21:20.138726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-model-cache\") pod \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") "
Jul 17 22:21:20.138726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138658    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-dshm\") pod \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") "
Jul 17 22:21:20.138726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tmp-dir\") pod \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") "
Jul 17 22:21:20.138901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7w9g\" (UniqueName: \"kubernetes.io/projected/49368892-14e6-43f6-b4e6-7a1f9dcd9923-kube-api-access-x7w9g\") pod \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") "
Jul 17 22:21:20.138901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tls-certs\") pod \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\" (UID: \"49368892-14e6-43f6-b4e6-7a1f9dcd9923\") "
Jul 17 22:21:20.138901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-home" (OuterVolumeSpecName: "home") pod "49368892-14e6-43f6-b4e6-7a1f9dcd9923" (UID: "49368892-14e6-43f6-b4e6-7a1f9dcd9923"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:20.139058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.138922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-model-cache" (OuterVolumeSpecName: "model-cache") pod "49368892-14e6-43f6-b4e6-7a1f9dcd9923" (UID: "49368892-14e6-43f6-b4e6-7a1f9dcd9923"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:20.139058 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.139000    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49368892-14e6-43f6-b4e6-7a1f9dcd9923" (UID: "49368892-14e6-43f6-b4e6-7a1f9dcd9923"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:20.139166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.139102    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:20.139166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.139120    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:20.139166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.139134    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:20.141017 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.140991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-dshm" (OuterVolumeSpecName: "dshm") pod "49368892-14e6-43f6-b4e6-7a1f9dcd9923" (UID: "49368892-14e6-43f6-b4e6-7a1f9dcd9923"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:21:20.141117 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.140991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49368892-14e6-43f6-b4e6-7a1f9dcd9923-kube-api-access-x7w9g" (OuterVolumeSpecName: "kube-api-access-x7w9g") pod "49368892-14e6-43f6-b4e6-7a1f9dcd9923" (UID: "49368892-14e6-43f6-b4e6-7a1f9dcd9923"). InnerVolumeSpecName "kube-api-access-x7w9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:21:20.141117 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.141025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49368892-14e6-43f6-b4e6-7a1f9dcd9923" (UID: "49368892-14e6-43f6-b4e6-7a1f9dcd9923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:21:20.239580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.239541    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49368892-14e6-43f6-b4e6-7a1f9dcd9923-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:20.239580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.239574    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7w9g\" (UniqueName: \"kubernetes.io/projected/49368892-14e6-43f6-b4e6-7a1f9dcd9923-kube-api-access-x7w9g\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:20.239580 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.239585    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49368892-14e6-43f6-b4e6-7a1f9dcd9923-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:20.310086 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.310001    2563 generic.go:356] "Generic (PLEG): container finished" podID="49368892-14e6-43f6-b4e6-7a1f9dcd9923" containerID="651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612" exitCode=0
Jul 17 22:21:20.310086 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.310051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"
Jul 17 22:21:20.310561 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.310054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm" event={"ID":"49368892-14e6-43f6-b4e6-7a1f9dcd9923","Type":"ContainerDied","Data":"651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612"}
Jul 17 22:21:20.310561 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.310144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm" event={"ID":"49368892-14e6-43f6-b4e6-7a1f9dcd9923","Type":"ContainerDied","Data":"32564b94f4d823d0f59f0c660e9090a8af9b231a12bee2a9bea631e190db19f2"}
Jul 17 22:21:20.310561 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.310161    2563 scope.go:117] "RemoveContainer" containerID="651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612"
Jul 17 22:21:20.323221 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.323202    2563 scope.go:117] "RemoveContainer" containerID="651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612"
Jul 17 22:21:20.323484 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:20.323465    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612\": container with ID starting with 651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612 not found: ID does not exist" containerID="651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612"
Jul 17 22:21:20.323544 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.323493    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612"} err="failed to get container status \"651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612\": rpc error: code = NotFound desc = could not find container \"651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612\": container with ID starting with 651dd8107e36f38c51124d2752ac512c79c08248cd7be6253863e0fd7e4cc612 not found: ID does not exist"
Jul 17 22:21:20.334532 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.334506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"]
Jul 17 22:21:20.340221 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.340198    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/sub-test-model-5401e5-kserve-84cdd8b6bc-4xmdm"]
Jul 17 22:21:20.604207 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.604127    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49368892-14e6-43f6-b4e6-7a1f9dcd9923" path="/var/lib/kubelet/pods/49368892-14e6-43f6-b4e6-7a1f9dcd9923/volumes"
Jul 17 22:21:20.604531 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:20.604517    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae8b611-14d9-48ff-8651-4358f85d03cf" path="/var/lib/kubelet/pods/5ae8b611-14d9-48ff-8651-4358f85d03cf/volumes"
Jul 17 22:21:25.597431 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:25.597397    2563 scope.go:117] "RemoveContainer" containerID="eb547748d78fef3c6481c45625202b95cd4faac13ba13f6ae3942e6591181767"
Jul 17 22:21:26.348898 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:26.348868    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerID="731405f3f8cfa532183ef9d066b5797f4970fbbaa54b9d5c24cdf1c6b05ec7a5" exitCode=6
Jul 17 22:21:26.349062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:26.348939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" event={"ID":"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef","Type":"ContainerDied","Data":"731405f3f8cfa532183ef9d066b5797f4970fbbaa54b9d5c24cdf1c6b05ec7a5"}
Jul 17 22:21:26.349062 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:26.349002    2563 scope.go:117] "RemoveContainer" containerID="eb547748d78fef3c6481c45625202b95cd4faac13ba13f6ae3942e6591181767"
Jul 17 22:21:26.349300 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:26.349278    2563 scope.go:117] "RemoveContainer" containerID="731405f3f8cfa532183ef9d066b5797f4970fbbaa54b9d5c24cdf1c6b05ec7a5"
Jul 17 22:21:26.349536 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:26.349510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8_odh-ai-gateway-infra(0ce4bcbb-443a-429e-96be-a9b6ecbe81ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef"
Jul 17 22:21:27.379696 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:27.378274    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"]
Jul 17 22:21:27.524760 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:27.524726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:27.700507 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:27.700479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef-maas-api-service-ca\") pod \"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef\" (UID: \"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef\") "
Jul 17 22:21:27.700883 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:27.700858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" (UID: "0ce4bcbb-443a-429e-96be-a9b6ecbe81ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:27.801286 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:27.801232    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:28.362609 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:28.362574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"
Jul 17 22:21:28.362807 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:28.362573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8" event={"ID":"0ce4bcbb-443a-429e-96be-a9b6ecbe81ef","Type":"ContainerDied","Data":"2469e4103c59b03519cbd46e7982d73da84055ed72d8f2c0007935cc47218427"}
Jul 17 22:21:28.362807 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:28.362717    2563 scope.go:117] "RemoveContainer" containerID="731405f3f8cfa532183ef9d066b5797f4970fbbaa54b9d5c24cdf1c6b05ec7a5"
Jul 17 22:21:28.383463 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:28.383438    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"]
Jul 17 22:21:28.389331 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:28.389307    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2pkp8"]
Jul 17 22:21:28.602923 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:28.602891    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" path="/var/lib/kubelet/pods/0ce4bcbb-443a-429e-96be-a9b6ecbe81ef/volumes"
Jul 17 22:21:30.503405 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503370    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"]
Jul 17 22:21:30.503806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503764    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:30.503806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503777    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:30.503806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503791    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae8b611-14d9-48ff-8651-4358f85d03cf" containerName="main"
Jul 17 22:21:30.503806 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503797    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae8b611-14d9-48ff-8651-4358f85d03cf" containerName="main"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503816    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49368892-14e6-43f6-b4e6-7a1f9dcd9923" containerName="main"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503822    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49368892-14e6-43f6-b4e6-7a1f9dcd9923" containerName="main"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503830    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503836    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae8b611-14d9-48ff-8651-4358f85d03cf" containerName="main"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503901    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503908    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49368892-14e6-43f6-b4e6-7a1f9dcd9923" containerName="main"
Jul 17 22:21:30.503941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.503915    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:30.508300 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.508283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:30.513730 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.513704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"]
Jul 17 22:21:30.630658 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.630628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/458add2d-b483-4ffd-8100-f1515f4b04d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6\" (UID: \"458add2d-b483-4ffd-8100-f1515f4b04d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:30.732094 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.732052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/458add2d-b483-4ffd-8100-f1515f4b04d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6\" (UID: \"458add2d-b483-4ffd-8100-f1515f4b04d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:30.732763 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.732738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/458add2d-b483-4ffd-8100-f1515f4b04d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6\" (UID: \"458add2d-b483-4ffd-8100-f1515f4b04d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:30.820535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.820426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:30.962229 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:30.962197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"]
Jul 17 22:21:30.966244 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:21:30.966203    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458add2d_b483_4ffd_8100_f1515f4b04d8.slice/crio-ef3227ccf1b629fd8e64922e06829e4bf7ed39d4f0e1bed1d4da919c91c39624 WatchSource:0}: Error finding container ef3227ccf1b629fd8e64922e06829e4bf7ed39d4f0e1bed1d4da919c91c39624: Status 404 returned error can't find the container with id ef3227ccf1b629fd8e64922e06829e4bf7ed39d4f0e1bed1d4da919c91c39624
Jul 17 22:21:31.381083 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:31.381048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" event={"ID":"458add2d-b483-4ffd-8100-f1515f4b04d8","Type":"ContainerStarted","Data":"65d11803e7c81c0b8a4cc59f8a88aa9d4a03bdd4f53ec878c7cb008f8a4097c2"}
Jul 17 22:21:31.381258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:31.381104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" event={"ID":"458add2d-b483-4ffd-8100-f1515f4b04d8","Type":"ContainerStarted","Data":"ef3227ccf1b629fd8e64922e06829e4bf7ed39d4f0e1bed1d4da919c91c39624"}
Jul 17 22:21:31.396817 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:31.396764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" podStartSLOduration=1.396749694 podStartE2EDuration="1.396749694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:21:31.395327599 +0000 UTC m=+3255.307567887" watchObservedRunningTime="2026-07-17 22:21:31.396749694 +0000 UTC m=+3255.308989983"
Jul 17 22:21:32.388279 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:32.388234    2563 generic.go:356] "Generic (PLEG): container finished" podID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerID="65d11803e7c81c0b8a4cc59f8a88aa9d4a03bdd4f53ec878c7cb008f8a4097c2" exitCode=6
Jul 17 22:21:32.388691 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:32.388322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" event={"ID":"458add2d-b483-4ffd-8100-f1515f4b04d8","Type":"ContainerDied","Data":"65d11803e7c81c0b8a4cc59f8a88aa9d4a03bdd4f53ec878c7cb008f8a4097c2"}
Jul 17 22:21:32.388691 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:32.388499    2563 scope.go:117] "RemoveContainer" containerID="65d11803e7c81c0b8a4cc59f8a88aa9d4a03bdd4f53ec878c7cb008f8a4097c2"
Jul 17 22:21:32.596785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:32.596753    2563 scope.go:117] "RemoveContainer" containerID="574e58a7128d9e20153ede471c594bd389f788f042285bc283158312112d292d"
Jul 17 22:21:33.395809 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.395779    2563 generic.go:356] "Generic (PLEG): container finished" podID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerID="334e9f6a75f13e4d7212441ffcad32a1983a61aa90d12b3e8ed52563d488dcb2" exitCode=6
Jul 17 22:21:33.396225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.395854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" event={"ID":"fbe33f76-6610-46e3-990c-b678fd2052f0","Type":"ContainerDied","Data":"334e9f6a75f13e4d7212441ffcad32a1983a61aa90d12b3e8ed52563d488dcb2"}
Jul 17 22:21:33.396225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.395941    2563 scope.go:117] "RemoveContainer" containerID="574e58a7128d9e20153ede471c594bd389f788f042285bc283158312112d292d"
Jul 17 22:21:33.396389 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.396289    2563 scope.go:117] "RemoveContainer" containerID="334e9f6a75f13e4d7212441ffcad32a1983a61aa90d12b3e8ed52563d488dcb2"
Jul 17 22:21:33.396532 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:33.396507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm_odh-ai-gateway-infra(fbe33f76-6610-46e3-990c-b678fd2052f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0"
Jul 17 22:21:33.397945 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.397922    2563 generic.go:356] "Generic (PLEG): container finished" podID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerID="fdf18139bdd09aebe46589adfa6396c9bc8c02920dcc840d96d6a060b76c5db0" exitCode=6
Jul 17 22:21:33.398038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.397987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" event={"ID":"458add2d-b483-4ffd-8100-f1515f4b04d8","Type":"ContainerDied","Data":"fdf18139bdd09aebe46589adfa6396c9bc8c02920dcc840d96d6a060b76c5db0"}
Jul 17 22:21:33.398205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.398189    2563 scope.go:117] "RemoveContainer" containerID="fdf18139bdd09aebe46589adfa6396c9bc8c02920dcc840d96d6a060b76c5db0"
Jul 17 22:21:33.398376 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:33.398357    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6_odh-ai-gateway-infra(458add2d-b483-4ffd-8100-f1515f4b04d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8"
Jul 17 22:21:33.408003 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.407987    2563 scope.go:117] "RemoveContainer" containerID="65d11803e7c81c0b8a4cc59f8a88aa9d4a03bdd4f53ec878c7cb008f8a4097c2"
Jul 17 22:21:33.926960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.926923    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"]
Jul 17 22:21:33.927306 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.927294    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:33.927364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.927308    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:33.927410 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.927399    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce4bcbb-443a-429e-96be-a9b6ecbe81ef" containerName="revoke-keys"
Jul 17 22:21:33.931869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.931851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:33.936054 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.936029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"]
Jul 17 22:21:33.962819 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:33.962791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f99cb92-d8ca-4f40-accc-ef1429eb6d93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9\" (UID: \"0f99cb92-d8ca-4f40-accc-ef1429eb6d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:34.063369 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.063339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f99cb92-d8ca-4f40-accc-ef1429eb6d93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9\" (UID: \"0f99cb92-d8ca-4f40-accc-ef1429eb6d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:34.063996 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.063977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f99cb92-d8ca-4f40-accc-ef1429eb6d93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9\" (UID: \"0f99cb92-d8ca-4f40-accc-ef1429eb6d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:34.109163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.109119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"]
Jul 17 22:21:34.113528 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.113505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.116264 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.116234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"kube-root-ca.crt\""
Jul 17 22:21:34.116416 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.116237    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"openshift-service-ca.crt\""
Jul 17 22:21:34.116540 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.116523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"default-dockercfg-g799l\""
Jul 17 22:21:34.116663 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.116560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8e6a3a\"/\"rate-test-model-8e6a3a-kserve-self-signed-certs\""
Jul 17 22:21:34.122839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.122814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"]
Jul 17 22:21:34.164305 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.164269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-model-cache\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.164305 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.164308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tmp-dir\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.164549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.164325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tls-certs\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.164549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.164422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84d5\" (UniqueName: \"kubernetes.io/projected/2ffb609c-52d2-4c50-91ee-02be94efb0d6-kube-api-access-g84d5\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.164549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.164456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-home\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.164549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.164512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-dshm\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.245387 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.245306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:34.265690 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.265660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-dshm\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.265836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.265712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-model-cache\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.265836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.265740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tmp-dir\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.265836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.265760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tls-certs\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.265836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.265822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84d5\" (UniqueName: \"kubernetes.io/projected/2ffb609c-52d2-4c50-91ee-02be94efb0d6-kube-api-access-g84d5\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.266074 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.265848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-home\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.266192 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.266166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-model-cache\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.266261 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.266191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tmp-dir\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.266446 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.266422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-home\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.268164 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.268131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-dshm\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.268469 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.268453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tls-certs\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.274976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.274946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84d5\" (UniqueName: \"kubernetes.io/projected/2ffb609c-52d2-4c50-91ee-02be94efb0d6-kube-api-access-g84d5\") pod \"rate-test-model-8e6a3a-kserve-b48757cff-x229w\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") " pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.389248 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.389218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"]
Jul 17 22:21:34.389995 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:21:34.389969    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f99cb92_d8ca_4f40_accc_ef1429eb6d93.slice/crio-69992a078b9b757dd963d7496160cb6cf911f7eaa8e4ff8d657bc45d50396f53 WatchSource:0}: Error finding container 69992a078b9b757dd963d7496160cb6cf911f7eaa8e4ff8d657bc45d50396f53: Status 404 returned error can't find the container with id 69992a078b9b757dd963d7496160cb6cf911f7eaa8e4ff8d657bc45d50396f53
Jul 17 22:21:34.410530 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.410502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" event={"ID":"0f99cb92-d8ca-4f40-accc-ef1429eb6d93","Type":"ContainerStarted","Data":"69992a078b9b757dd963d7496160cb6cf911f7eaa8e4ff8d657bc45d50396f53"}
Jul 17 22:21:34.412256 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.412231    2563 scope.go:117] "RemoveContainer" containerID="fdf18139bdd09aebe46589adfa6396c9bc8c02920dcc840d96d6a060b76c5db0"
Jul 17 22:21:34.412450 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:34.412431    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6_odh-ai-gateway-infra(458add2d-b483-4ffd-8100-f1515f4b04d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8"
Jul 17 22:21:34.421181 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.421159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"]
Jul 17 22:21:34.423734 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.423714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:34.570451 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.570424    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:34.669551 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.669520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe33f76-6610-46e3-990c-b678fd2052f0-maas-api-service-ca\") pod \"fbe33f76-6610-46e3-990c-b678fd2052f0\" (UID: \"fbe33f76-6610-46e3-990c-b678fd2052f0\") "
Jul 17 22:21:34.669922 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.669900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe33f76-6610-46e3-990c-b678fd2052f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbe33f76-6610-46e3-990c-b678fd2052f0" (UID: "fbe33f76-6610-46e3-990c-b678fd2052f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:34.770548 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.770482    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe33f76-6610-46e3-990c-b678fd2052f0-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:34.784633 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:34.784585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"]
Jul 17 22:21:34.786402 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:21:34.786376    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ffb609c_52d2_4c50_91ee_02be94efb0d6.slice/crio-568fbe425b1eb5723829e7a19d42554e6abd4ffc275d3c9ef59816cc825f3bff WatchSource:0}: Error finding container 568fbe425b1eb5723829e7a19d42554e6abd4ffc275d3c9ef59816cc825f3bff: Status 404 returned error can't find the container with id 568fbe425b1eb5723829e7a19d42554e6abd4ffc275d3c9ef59816cc825f3bff
Jul 17 22:21:35.418565 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.418537    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"
Jul 17 22:21:35.418999 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.418574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm" event={"ID":"fbe33f76-6610-46e3-990c-b678fd2052f0","Type":"ContainerDied","Data":"7a924b82ca645ef950480e582be4c2cc9856e16283e8c10b6570175512175a3c"}
Jul 17 22:21:35.418999 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.418667    2563 scope.go:117] "RemoveContainer" containerID="334e9f6a75f13e4d7212441ffcad32a1983a61aa90d12b3e8ed52563d488dcb2"
Jul 17 22:21:35.420403 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.420379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w" event={"ID":"2ffb609c-52d2-4c50-91ee-02be94efb0d6","Type":"ContainerStarted","Data":"9f853dfa6aef6996d4937c45d761b3784e693c022dc2bc4b5751dc279b4e98e0"}
Jul 17 22:21:35.420520 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.420418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w" event={"ID":"2ffb609c-52d2-4c50-91ee-02be94efb0d6","Type":"ContainerStarted","Data":"568fbe425b1eb5723829e7a19d42554e6abd4ffc275d3c9ef59816cc825f3bff"}
Jul 17 22:21:35.422156 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.422124    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerID="3356c30317b90f0cef49296f17a1bad7371f6b74bd537be8a84189744ecee3e8" exitCode=6
Jul 17 22:21:35.422272 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.422151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" event={"ID":"0f99cb92-d8ca-4f40-accc-ef1429eb6d93","Type":"ContainerDied","Data":"3356c30317b90f0cef49296f17a1bad7371f6b74bd537be8a84189744ecee3e8"}
Jul 17 22:21:35.422387 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.422371    2563 scope.go:117] "RemoveContainer" containerID="3356c30317b90f0cef49296f17a1bad7371f6b74bd537be8a84189744ecee3e8"
Jul 17 22:21:35.442843 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.442797    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w" podStartSLOduration=1.442782383 podStartE2EDuration="1.442782383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:21:35.441125913 +0000 UTC m=+3259.353366203" watchObservedRunningTime="2026-07-17 22:21:35.442782383 +0000 UTC m=+3259.355022666"
Jul 17 22:21:35.467699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.467673    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"]
Jul 17 22:21:35.469706 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:35.469679    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tpzrm"]
Jul 17 22:21:36.430862 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:36.430828    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerID="cbf2848dfacbdd31278bcecb8b5ff44fa25857db0618d04efaa245951a4b18cc" exitCode=6
Jul 17 22:21:36.431308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:36.430918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" event={"ID":"0f99cb92-d8ca-4f40-accc-ef1429eb6d93","Type":"ContainerDied","Data":"cbf2848dfacbdd31278bcecb8b5ff44fa25857db0618d04efaa245951a4b18cc"}
Jul 17 22:21:36.431308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:36.430990    2563 scope.go:117] "RemoveContainer" containerID="3356c30317b90f0cef49296f17a1bad7371f6b74bd537be8a84189744ecee3e8"
Jul 17 22:21:36.431308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:36.431207    2563 scope.go:117] "RemoveContainer" containerID="cbf2848dfacbdd31278bcecb8b5ff44fa25857db0618d04efaa245951a4b18cc"
Jul 17 22:21:36.431453 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:36.431428    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9_odh-ai-gateway-infra(0f99cb92-d8ca-4f40-accc-ef1429eb6d93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93"
Jul 17 22:21:36.605870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:36.605824    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" path="/var/lib/kubelet/pods/fbe33f76-6610-46e3-990c-b678fd2052f0/volumes"
Jul 17 22:21:37.437836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:37.437813    2563 scope.go:117] "RemoveContainer" containerID="cbf2848dfacbdd31278bcecb8b5ff44fa25857db0618d04efaa245951a4b18cc"
Jul 17 22:21:37.438305 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:37.438097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9_odh-ai-gateway-infra(0f99cb92-d8ca-4f40-accc-ef1429eb6d93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93"
Jul 17 22:21:44.424057 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:44.424021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:44.424485 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:44.424094    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:44.436635 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:44.436607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:44.486714 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:44.486682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:21:44.596730 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:44.596700    2563 scope.go:117] "RemoveContainer" containerID="fdf18139bdd09aebe46589adfa6396c9bc8c02920dcc840d96d6a060b76c5db0"
Jul 17 22:21:45.482684 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:45.482650    2563 generic.go:356] "Generic (PLEG): container finished" podID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerID="427421f33d09b3df94ba0ddd10703f9d22cecc7121e1a6ba831f3bbfc54dc864" exitCode=6
Jul 17 22:21:45.483067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:45.482724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" event={"ID":"458add2d-b483-4ffd-8100-f1515f4b04d8","Type":"ContainerDied","Data":"427421f33d09b3df94ba0ddd10703f9d22cecc7121e1a6ba831f3bbfc54dc864"}
Jul 17 22:21:45.483067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:45.482784    2563 scope.go:117] "RemoveContainer" containerID="fdf18139bdd09aebe46589adfa6396c9bc8c02920dcc840d96d6a060b76c5db0"
Jul 17 22:21:45.483067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:45.483030    2563 scope.go:117] "RemoveContainer" containerID="427421f33d09b3df94ba0ddd10703f9d22cecc7121e1a6ba831f3bbfc54dc864"
Jul 17 22:21:45.483264 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:45.483243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6_odh-ai-gateway-infra(458add2d-b483-4ffd-8100-f1515f4b04d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8"
Jul 17 22:21:46.516348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:46.516309    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"]
Jul 17 22:21:46.665159 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:46.665130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:46.775874 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:46.775793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/458add2d-b483-4ffd-8100-f1515f4b04d8-maas-api-service-ca\") pod \"458add2d-b483-4ffd-8100-f1515f4b04d8\" (UID: \"458add2d-b483-4ffd-8100-f1515f4b04d8\") "
Jul 17 22:21:46.776144 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:46.776122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458add2d-b483-4ffd-8100-f1515f4b04d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "458add2d-b483-4ffd-8100-f1515f4b04d8" (UID: "458add2d-b483-4ffd-8100-f1515f4b04d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:46.876726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:46.876695    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/458add2d-b483-4ffd-8100-f1515f4b04d8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:47.494726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:47.494690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"
Jul 17 22:21:47.495025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:47.494730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6" event={"ID":"458add2d-b483-4ffd-8100-f1515f4b04d8","Type":"ContainerDied","Data":"ef3227ccf1b629fd8e64922e06829e4bf7ed39d4f0e1bed1d4da919c91c39624"}
Jul 17 22:21:47.495025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:47.494792    2563 scope.go:117] "RemoveContainer" containerID="427421f33d09b3df94ba0ddd10703f9d22cecc7121e1a6ba831f3bbfc54dc864"
Jul 17 22:21:47.517624 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:47.517578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"]
Jul 17 22:21:47.526129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:47.526107    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-cp5q6"]
Jul 17 22:21:48.603348 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:48.603312    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" path="/var/lib/kubelet/pods/458add2d-b483-4ffd-8100-f1515f4b04d8/volumes"
Jul 17 22:21:51.597083 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:51.597056    2563 scope.go:117] "RemoveContainer" containerID="cbf2848dfacbdd31278bcecb8b5ff44fa25857db0618d04efaa245951a4b18cc"
Jul 17 22:21:52.522528 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:52.522496    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerID="97566593a6a5d6980364a97c17666c1ef032484d329934ca401bf1b3cbb33b57" exitCode=6
Jul 17 22:21:52.522747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:52.522579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" event={"ID":"0f99cb92-d8ca-4f40-accc-ef1429eb6d93","Type":"ContainerDied","Data":"97566593a6a5d6980364a97c17666c1ef032484d329934ca401bf1b3cbb33b57"}
Jul 17 22:21:52.522747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:52.522681    2563 scope.go:117] "RemoveContainer" containerID="cbf2848dfacbdd31278bcecb8b5ff44fa25857db0618d04efaa245951a4b18cc"
Jul 17 22:21:52.523029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:52.522998    2563 scope.go:117] "RemoveContainer" containerID="97566593a6a5d6980364a97c17666c1ef032484d329934ca401bf1b3cbb33b57"
Jul 17 22:21:52.523249 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:21:52.523231    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9_odh-ai-gateway-infra(0f99cb92-d8ca-4f40-accc-ef1429eb6d93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93"
Jul 17 22:21:53.548728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:53.548691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"]
Jul 17 22:21:53.690078 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:53.690046    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:53.740132 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:53.740092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f99cb92-d8ca-4f40-accc-ef1429eb6d93-maas-api-service-ca\") pod \"0f99cb92-d8ca-4f40-accc-ef1429eb6d93\" (UID: \"0f99cb92-d8ca-4f40-accc-ef1429eb6d93\") "
Jul 17 22:21:53.740501 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:53.740481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f99cb92-d8ca-4f40-accc-ef1429eb6d93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f99cb92-d8ca-4f40-accc-ef1429eb6d93" (UID: "0f99cb92-d8ca-4f40-accc-ef1429eb6d93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:53.841935 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:53.841849    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f99cb92-d8ca-4f40-accc-ef1429eb6d93-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:21:54.535877 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:54.535839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9" event={"ID":"0f99cb92-d8ca-4f40-accc-ef1429eb6d93","Type":"ContainerDied","Data":"69992a078b9b757dd963d7496160cb6cf911f7eaa8e4ff8d657bc45d50396f53"}
Jul 17 22:21:54.535877 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:54.535872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"
Jul 17 22:21:54.536148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:54.535900    2563 scope.go:117] "RemoveContainer" containerID="97566593a6a5d6980364a97c17666c1ef032484d329934ca401bf1b3cbb33b57"
Jul 17 22:21:54.561836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:54.561803    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"]
Jul 17 22:21:54.566383 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:54.566357    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-7s6k9"]
Jul 17 22:21:54.602391 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:54.602350    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" path="/var/lib/kubelet/pods/0f99cb92-d8ca-4f40-accc-ef1429eb6d93/volumes"
Jul 17 22:21:56.117941 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.117910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"]
Jul 17 22:21:56.118457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118435    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.118457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118467    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118473    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118485    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118491    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118505    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118517    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118560    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.118591 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118659    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118665    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118671    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118683    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe33f76-6610-46e3-990c-b678fd2052f0" containerName="revoke-keys"
Jul 17 22:21:56.119193 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.118695    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f99cb92-d8ca-4f40-accc-ef1429eb6d93" containerName="revoke-keys"
Jul 17 22:21:56.123172 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.123155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.125558 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.125525    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"openshift-service-ca.crt\""
Jul 17 22:21:56.125688 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.125633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"kube-root-ca.crt\""
Jul 17 22:21:56.126528 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.126513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"default-dockercfg-99xqt\""
Jul 17 22:21:56.126609 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.126569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5401e5\"/\"rate-test-model-5401e5-kserve-self-signed-certs\""
Jul 17 22:21:56.132213 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.132189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"]
Jul 17 22:21:56.164549 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.164512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-tmp-dir\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.164750 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.164586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aded8163-d299-4859-9d00-8a48abe4c3a6-tls-certs\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.164750 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.164710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-home\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.164750 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.164745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-dshm\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.164882 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.164768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-model-cache\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.164882 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.164791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbgr\" (UniqueName: \"kubernetes.io/projected/aded8163-d299-4859-9d00-8a48abe4c3a6-kube-api-access-nzbgr\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aded8163-d299-4859-9d00-8a48abe4c3a6-tls-certs\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266138 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-home\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266365 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-dshm\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266365 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-model-cache\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266365 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbgr\" (UniqueName: \"kubernetes.io/projected/aded8163-d299-4859-9d00-8a48abe4c3a6-kube-api-access-nzbgr\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-tmp-dir\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266573 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-home\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266657 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-model-cache\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.266761 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.266741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-tmp-dir\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.268752 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.268731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-dshm\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.268937 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.268921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aded8163-d299-4859-9d00-8a48abe4c3a6-tls-certs\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.274778 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.274751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbgr\" (UniqueName: \"kubernetes.io/projected/aded8163-d299-4859-9d00-8a48abe4c3a6-kube-api-access-nzbgr\") pod \"rate-test-model-5401e5-kserve-559c97b57f-c6rnk\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") " pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.434320 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.434287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:21:56.581346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:56.581318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"]
Jul 17 22:21:56.582758 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:21:56.582726    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaded8163_d299_4859_9d00_8a48abe4c3a6.slice/crio-ce1657007f14b859d53ac52b3c8b4ac19d7b505c186aefbc304e2bcb65231b59 WatchSource:0}: Error finding container ce1657007f14b859d53ac52b3c8b4ac19d7b505c186aefbc304e2bcb65231b59: Status 404 returned error can't find the container with id ce1657007f14b859d53ac52b3c8b4ac19d7b505c186aefbc304e2bcb65231b59
Jul 17 22:21:57.554687 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:57.554643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk" event={"ID":"aded8163-d299-4859-9d00-8a48abe4c3a6","Type":"ContainerStarted","Data":"9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02"}
Jul 17 22:21:57.555069 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:57.554700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk" event={"ID":"aded8163-d299-4859-9d00-8a48abe4c3a6","Type":"ContainerStarted","Data":"ce1657007f14b859d53ac52b3c8b4ac19d7b505c186aefbc304e2bcb65231b59"}
Jul 17 22:21:57.573304 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:21:57.573251    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk" podStartSLOduration=1.5732346480000001 podStartE2EDuration="1.573234648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:21:57.570528591 +0000 UTC m=+3281.482768879" watchObservedRunningTime="2026-07-17 22:21:57.573234648 +0000 UTC m=+3281.485474937"
Jul 17 22:22:06.231347 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.231305    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"]
Jul 17 22:22:06.231733 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.231679    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:22:06.231733 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.231689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:22:06.231822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.231768    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="458add2d-b483-4ffd-8100-f1515f4b04d8" containerName="revoke-keys"
Jul 17 22:22:06.234945 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.234926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:06.237392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.237368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:22:06.241330 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.241304    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"]
Jul 17 22:22:06.357240 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.357196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fc1a1-7006-491d-bc54-49cd150f741d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw\" (UID: \"232fc1a1-7006-491d-bc54-49cd150f741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:06.435087 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.435049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:22:06.435267 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.435134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:22:06.447730 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.447701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:22:06.458740 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.458716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fc1a1-7006-491d-bc54-49cd150f741d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw\" (UID: \"232fc1a1-7006-491d-bc54-49cd150f741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:06.459333 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.459310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fc1a1-7006-491d-bc54-49cd150f741d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw\" (UID: \"232fc1a1-7006-491d-bc54-49cd150f741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:06.547146 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.547067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:06.627379 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.627348    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:22:06.700272 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:06.700234    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"]
Jul 17 22:22:06.702939 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:22:06.702900    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232fc1a1_7006_491d_bc54_49cd150f741d.slice/crio-93fb70900e91d59c3fe040a036b5e8f8a1f73720e02a940b2dc74997c5be2aff WatchSource:0}: Error finding container 93fb70900e91d59c3fe040a036b5e8f8a1f73720e02a940b2dc74997c5be2aff: Status 404 returned error can't find the container with id 93fb70900e91d59c3fe040a036b5e8f8a1f73720e02a940b2dc74997c5be2aff
Jul 17 22:22:07.620084 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:07.619987    2563 generic.go:356] "Generic (PLEG): container finished" podID="232fc1a1-7006-491d-bc54-49cd150f741d" containerID="0385730f4b655058c497e9073595a59f873cddf029d484eaffcb8fb21cb7a2a4" exitCode=6
Jul 17 22:22:07.620486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:07.620076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" event={"ID":"232fc1a1-7006-491d-bc54-49cd150f741d","Type":"ContainerDied","Data":"0385730f4b655058c497e9073595a59f873cddf029d484eaffcb8fb21cb7a2a4"}
Jul 17 22:22:07.620486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:07.620136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" event={"ID":"232fc1a1-7006-491d-bc54-49cd150f741d","Type":"ContainerStarted","Data":"93fb70900e91d59c3fe040a036b5e8f8a1f73720e02a940b2dc74997c5be2aff"}
Jul 17 22:22:07.620486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:07.620279    2563 scope.go:117] "RemoveContainer" containerID="0385730f4b655058c497e9073595a59f873cddf029d484eaffcb8fb21cb7a2a4"
Jul 17 22:22:08.316870 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.316840    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"]
Jul 17 22:22:08.320169 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.320153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:08.327987 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.327956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"]
Jul 17 22:22:08.478347 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.478316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912b9eb0-f756-42d2-bb15-a02f0d4f25bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw\" (UID: \"912b9eb0-f756-42d2-bb15-a02f0d4f25bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:08.579109 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.579026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912b9eb0-f756-42d2-bb15-a02f0d4f25bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw\" (UID: \"912b9eb0-f756-42d2-bb15-a02f0d4f25bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:08.579713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.579693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912b9eb0-f756-42d2-bb15-a02f0d4f25bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw\" (UID: \"912b9eb0-f756-42d2-bb15-a02f0d4f25bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:08.626586 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.626543    2563 generic.go:356] "Generic (PLEG): container finished" podID="232fc1a1-7006-491d-bc54-49cd150f741d" containerID="89e82c293e60ff0641a56a4e8e3a941694ce7d6b95c33a18b5c73108c3b3232e" exitCode=6
Jul 17 22:22:08.627025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.626630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" event={"ID":"232fc1a1-7006-491d-bc54-49cd150f741d","Type":"ContainerDied","Data":"89e82c293e60ff0641a56a4e8e3a941694ce7d6b95c33a18b5c73108c3b3232e"}
Jul 17 22:22:08.627025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.626699    2563 scope.go:117] "RemoveContainer" containerID="0385730f4b655058c497e9073595a59f873cddf029d484eaffcb8fb21cb7a2a4"
Jul 17 22:22:08.627025 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.626946    2563 scope.go:117] "RemoveContainer" containerID="89e82c293e60ff0641a56a4e8e3a941694ce7d6b95c33a18b5c73108c3b3232e"
Jul 17 22:22:08.627186 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:08.627167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw_odh-ai-gateway-infra(232fc1a1-7006-491d-bc54-49cd150f741d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" podUID="232fc1a1-7006-491d-bc54-49cd150f741d"
Jul 17 22:22:08.638963 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.638945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:08.786089 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:08.786065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"]
Jul 17 22:22:08.787936 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:22:08.787911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912b9eb0_f756_42d2_bb15_a02f0d4f25bc.slice/crio-5318a2e987d470d81795daf3b132b40f03911df0c57fcfaa4d0b5a01ebdc31d3 WatchSource:0}: Error finding container 5318a2e987d470d81795daf3b132b40f03911df0c57fcfaa4d0b5a01ebdc31d3: Status 404 returned error can't find the container with id 5318a2e987d470d81795daf3b132b40f03911df0c57fcfaa4d0b5a01ebdc31d3
Jul 17 22:22:09.633793 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:09.633708    2563 scope.go:117] "RemoveContainer" containerID="89e82c293e60ff0641a56a4e8e3a941694ce7d6b95c33a18b5c73108c3b3232e"
Jul 17 22:22:09.634204 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:09.633927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw_odh-ai-gateway-infra(232fc1a1-7006-491d-bc54-49cd150f741d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" podUID="232fc1a1-7006-491d-bc54-49cd150f741d"
Jul 17 22:22:09.635120 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:09.635098    2563 generic.go:356] "Generic (PLEG): container finished" podID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerID="0ece5288ddb12cdb27c0d98101017878d3980b576428158557d32cafec2d5608" exitCode=6
Jul 17 22:22:09.635194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:09.635163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" event={"ID":"912b9eb0-f756-42d2-bb15-a02f0d4f25bc","Type":"ContainerDied","Data":"0ece5288ddb12cdb27c0d98101017878d3980b576428158557d32cafec2d5608"}
Jul 17 22:22:09.635234 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:09.635193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" event={"ID":"912b9eb0-f756-42d2-bb15-a02f0d4f25bc","Type":"ContainerStarted","Data":"5318a2e987d470d81795daf3b132b40f03911df0c57fcfaa4d0b5a01ebdc31d3"}
Jul 17 22:22:09.635293 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:09.635280    2563 scope.go:117] "RemoveContainer" containerID="0ece5288ddb12cdb27c0d98101017878d3980b576428158557d32cafec2d5608"
Jul 17 22:22:10.642271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:10.642182    2563 generic.go:356] "Generic (PLEG): container finished" podID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerID="02b6c7cfff62c99ce8cfe882bc320041be6fb932824c9420798e566a176acb5b" exitCode=6
Jul 17 22:22:10.642271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:10.642241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" event={"ID":"912b9eb0-f756-42d2-bb15-a02f0d4f25bc","Type":"ContainerDied","Data":"02b6c7cfff62c99ce8cfe882bc320041be6fb932824c9420798e566a176acb5b"}
Jul 17 22:22:10.642791 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:10.642300    2563 scope.go:117] "RemoveContainer" containerID="0ece5288ddb12cdb27c0d98101017878d3980b576428158557d32cafec2d5608"
Jul 17 22:22:10.642791 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:10.642493    2563 scope.go:117] "RemoveContainer" containerID="02b6c7cfff62c99ce8cfe882bc320041be6fb932824c9420798e566a176acb5b"
Jul 17 22:22:10.642791 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:10.642735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw_odh-ai-gateway-infra(912b9eb0-f756-42d2-bb15-a02f0d4f25bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc"
Jul 17 22:22:11.649362 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:11.649333    2563 scope.go:117] "RemoveContainer" containerID="02b6c7cfff62c99ce8cfe882bc320041be6fb932824c9420798e566a176acb5b"
Jul 17 22:22:11.649799 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:11.649507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw_odh-ai-gateway-infra(912b9eb0-f756-42d2-bb15-a02f0d4f25bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc"
Jul 17 22:22:16.818651 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:16.818626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:22:16.830841 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:16.830821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:22:20.546914 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.546877    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"]
Jul 17 22:22:20.551829 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.551810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:20.557169 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.557138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"]
Jul 17 22:22:20.692823 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.692787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4990d03-55db-4e94-94d2-036c98228da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w\" (UID: \"b4990d03-55db-4e94-94d2-036c98228da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:20.794110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.794074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4990d03-55db-4e94-94d2-036c98228da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w\" (UID: \"b4990d03-55db-4e94-94d2-036c98228da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:20.794824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.794802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4990d03-55db-4e94-94d2-036c98228da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w\" (UID: \"b4990d03-55db-4e94-94d2-036c98228da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:20.864063 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:20.864001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:21.231905 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:21.231714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"]
Jul 17 22:22:21.234172 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:22:21.234144    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4990d03_55db_4e94_94d2_036c98228da6.slice/crio-4e60cf3c76a48db1c04a568d2d9549849bac0c76ae2602aab5db98cba888492e WatchSource:0}: Error finding container 4e60cf3c76a48db1c04a568d2d9549849bac0c76ae2602aab5db98cba888492e: Status 404 returned error can't find the container with id 4e60cf3c76a48db1c04a568d2d9549849bac0c76ae2602aab5db98cba888492e
Jul 17 22:22:21.596483 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:21.596460    2563 scope.go:117] "RemoveContainer" containerID="89e82c293e60ff0641a56a4e8e3a941694ce7d6b95c33a18b5c73108c3b3232e"
Jul 17 22:22:21.704295 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:21.704264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" event={"ID":"b4990d03-55db-4e94-94d2-036c98228da6","Type":"ContainerStarted","Data":"e4ebb7eab7127862d391f82a6856e6fc64f3c42aa17b02493866887a8cc2edec"}
Jul 17 22:22:21.704456 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:21.704323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" event={"ID":"b4990d03-55db-4e94-94d2-036c98228da6","Type":"ContainerStarted","Data":"4e60cf3c76a48db1c04a568d2d9549849bac0c76ae2602aab5db98cba888492e"}
Jul 17 22:22:21.720940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:21.720889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" podStartSLOduration=1.720875997 podStartE2EDuration="1.720875997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:22:21.71902531 +0000 UTC m=+3305.631265601" watchObservedRunningTime="2026-07-17 22:22:21.720875997 +0000 UTC m=+3305.633116285"
Jul 17 22:22:22.711633 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.711579    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4990d03-55db-4e94-94d2-036c98228da6" containerID="e4ebb7eab7127862d391f82a6856e6fc64f3c42aa17b02493866887a8cc2edec" exitCode=6
Jul 17 22:22:22.712069 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.711671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" event={"ID":"b4990d03-55db-4e94-94d2-036c98228da6","Type":"ContainerDied","Data":"e4ebb7eab7127862d391f82a6856e6fc64f3c42aa17b02493866887a8cc2edec"}
Jul 17 22:22:22.712069 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.711859    2563 scope.go:117] "RemoveContainer" containerID="e4ebb7eab7127862d391f82a6856e6fc64f3c42aa17b02493866887a8cc2edec"
Jul 17 22:22:22.713503 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.713483    2563 generic.go:356] "Generic (PLEG): container finished" podID="232fc1a1-7006-491d-bc54-49cd150f741d" containerID="d35676a4211a6693a8c9082bd1844572f7245889bcc6cc8e7ab74adda5d7aa06" exitCode=6
Jul 17 22:22:22.713618 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.713551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" event={"ID":"232fc1a1-7006-491d-bc54-49cd150f741d","Type":"ContainerDied","Data":"d35676a4211a6693a8c9082bd1844572f7245889bcc6cc8e7ab74adda5d7aa06"}
Jul 17 22:22:22.713673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.713621    2563 scope.go:117] "RemoveContainer" containerID="89e82c293e60ff0641a56a4e8e3a941694ce7d6b95c33a18b5c73108c3b3232e"
Jul 17 22:22:22.713853 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:22.713832    2563 scope.go:117] "RemoveContainer" containerID="d35676a4211a6693a8c9082bd1844572f7245889bcc6cc8e7ab74adda5d7aa06"
Jul 17 22:22:22.714038 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:22.714013    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw_odh-ai-gateway-infra(232fc1a1-7006-491d-bc54-49cd150f741d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" podUID="232fc1a1-7006-491d-bc54-49cd150f741d"
Jul 17 22:22:23.721235 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.721198    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4990d03-55db-4e94-94d2-036c98228da6" containerID="02b01d64ce8289b21880b1e61810d3a28cecc865e3d761c986e92e773d5408e6" exitCode=6
Jul 17 22:22:23.721713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.721294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" event={"ID":"b4990d03-55db-4e94-94d2-036c98228da6","Type":"ContainerDied","Data":"02b01d64ce8289b21880b1e61810d3a28cecc865e3d761c986e92e773d5408e6"}
Jul 17 22:22:23.721713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.721379    2563 scope.go:117] "RemoveContainer" containerID="e4ebb7eab7127862d391f82a6856e6fc64f3c42aa17b02493866887a8cc2edec"
Jul 17 22:22:23.721713 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.721471    2563 scope.go:117] "RemoveContainer" containerID="02b01d64ce8289b21880b1e61810d3a28cecc865e3d761c986e92e773d5408e6"
Jul 17 22:22:23.721713 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:23.721690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w_odh-ai-gateway-infra(b4990d03-55db-4e94-94d2-036c98228da6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" podUID="b4990d03-55db-4e94-94d2-036c98228da6"
Jul 17 22:22:23.749669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.749642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"]
Jul 17 22:22:23.894314 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.894291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:23.918257 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.918226    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fc1a1-7006-491d-bc54-49cd150f741d-maas-api-service-ca\") pod \"232fc1a1-7006-491d-bc54-49cd150f741d\" (UID: \"232fc1a1-7006-491d-bc54-49cd150f741d\") "
Jul 17 22:22:23.918571 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:23.918546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232fc1a1-7006-491d-bc54-49cd150f741d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "232fc1a1-7006-491d-bc54-49cd150f741d" (UID: "232fc1a1-7006-491d-bc54-49cd150f741d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:24.019063 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.018975    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/232fc1a1-7006-491d-bc54-49cd150f741d-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:22:24.730335 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.730305    2563 scope.go:117] "RemoveContainer" containerID="02b01d64ce8289b21880b1e61810d3a28cecc865e3d761c986e92e773d5408e6"
Jul 17 22:22:24.730787 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:24.730567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w_odh-ai-gateway-infra(b4990d03-55db-4e94-94d2-036c98228da6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" podUID="b4990d03-55db-4e94-94d2-036c98228da6"
Jul 17 22:22:24.731364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.731347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"
Jul 17 22:22:24.731452 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.731389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw" event={"ID":"232fc1a1-7006-491d-bc54-49cd150f741d","Type":"ContainerDied","Data":"93fb70900e91d59c3fe040a036b5e8f8a1f73720e02a940b2dc74997c5be2aff"}
Jul 17 22:22:24.731452 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.731447    2563 scope.go:117] "RemoveContainer" containerID="d35676a4211a6693a8c9082bd1844572f7245889bcc6cc8e7ab74adda5d7aa06"
Jul 17 22:22:24.758483 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.758451    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"]
Jul 17 22:22:24.764046 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:24.764018    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-tgtvw"]
Jul 17 22:22:26.599878 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:26.599848    2563 scope.go:117] "RemoveContainer" containerID="02b6c7cfff62c99ce8cfe882bc320041be6fb932824c9420798e566a176acb5b"
Jul 17 22:22:26.603884 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:26.603858    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" path="/var/lib/kubelet/pods/232fc1a1-7006-491d-bc54-49cd150f741d/volumes"
Jul 17 22:22:27.753827 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:27.753787    2563 generic.go:356] "Generic (PLEG): container finished" podID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerID="0a86be7828b5edd6a47db098514de4e940eed5dbab1d8129c2694582eb356fd7" exitCode=6
Jul 17 22:22:27.754279 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:27.753852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" event={"ID":"912b9eb0-f756-42d2-bb15-a02f0d4f25bc","Type":"ContainerDied","Data":"0a86be7828b5edd6a47db098514de4e940eed5dbab1d8129c2694582eb356fd7"}
Jul 17 22:22:27.754279 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:27.753946    2563 scope.go:117] "RemoveContainer" containerID="02b6c7cfff62c99ce8cfe882bc320041be6fb932824c9420798e566a176acb5b"
Jul 17 22:22:27.754279 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:27.754254    2563 scope.go:117] "RemoveContainer" containerID="0a86be7828b5edd6a47db098514de4e940eed5dbab1d8129c2694582eb356fd7"
Jul 17 22:22:27.754508 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:27.754486    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw_odh-ai-gateway-infra(912b9eb0-f756-42d2-bb15-a02f0d4f25bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc"
Jul 17 22:22:28.780097 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:28.780065    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"]
Jul 17 22:22:28.933165 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:28.933139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:28.960447 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:28.960417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912b9eb0-f756-42d2-bb15-a02f0d4f25bc-maas-api-service-ca\") pod \"912b9eb0-f756-42d2-bb15-a02f0d4f25bc\" (UID: \"912b9eb0-f756-42d2-bb15-a02f0d4f25bc\") "
Jul 17 22:22:28.960770 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:28.960747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912b9eb0-f756-42d2-bb15-a02f0d4f25bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "912b9eb0-f756-42d2-bb15-a02f0d4f25bc" (UID: "912b9eb0-f756-42d2-bb15-a02f0d4f25bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:29.061315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:29.061228    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912b9eb0-f756-42d2-bb15-a02f0d4f25bc-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:22:29.767940 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:29.767916    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"
Jul 17 22:22:29.768146 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:29.767911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw" event={"ID":"912b9eb0-f756-42d2-bb15-a02f0d4f25bc","Type":"ContainerDied","Data":"5318a2e987d470d81795daf3b132b40f03911df0c57fcfaa4d0b5a01ebdc31d3"}
Jul 17 22:22:29.768146 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:29.768048    2563 scope.go:117] "RemoveContainer" containerID="0a86be7828b5edd6a47db098514de4e940eed5dbab1d8129c2694582eb356fd7"
Jul 17 22:22:29.788483 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:29.788451    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"]
Jul 17 22:22:29.793258 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:29.793234    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-mf5jw"]
Jul 17 22:22:30.603180 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:30.603145    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" path="/var/lib/kubelet/pods/912b9eb0-f756-42d2-bb15-a02f0d4f25bc/volumes"
Jul 17 22:22:33.971483 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.971447    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"]
Jul 17 22:22:33.972043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972023    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972067    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972084    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972093    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972111    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972131 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972119    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972148    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972157    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972261    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972278    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.972499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.972308    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:33.977026 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.977004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:33.981546 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:33.981521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"]
Jul 17 22:22:34.005759 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.005730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb\" (UID: \"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:34.107032 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.106993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb\" (UID: \"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:34.107646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.107626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb\" (UID: \"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:34.289483 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.289377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:34.428961 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.428861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"]
Jul 17 22:22:34.431336 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:22:34.431301    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5f2521_0a2d_4b55_a8f8_b342d8dd7491.slice/crio-cf9e0d19f1773724b37cfe67d4d0f05ef43c09eb72850038cda836ff72d878ea WatchSource:0}: Error finding container cf9e0d19f1773724b37cfe67d4d0f05ef43c09eb72850038cda836ff72d878ea: Status 404 returned error can't find the container with id cf9e0d19f1773724b37cfe67d4d0f05ef43c09eb72850038cda836ff72d878ea
Jul 17 22:22:34.798154 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.798114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerStarted","Data":"a128bcb14c41cc0de2c5b26f69460620ebff9b3c286230a9a552833844bd6c0e"}
Jul 17 22:22:34.798341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.798166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerStarted","Data":"cf9e0d19f1773724b37cfe67d4d0f05ef43c09eb72850038cda836ff72d878ea"}
Jul 17 22:22:34.816202 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:34.816156    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" podStartSLOduration=1.816141722 podStartE2EDuration="1.816141722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:22:34.813571853 +0000 UTC m=+3318.725812143" watchObservedRunningTime="2026-07-17 22:22:34.816141722 +0000 UTC m=+3318.728382010"
Jul 17 22:22:35.805403 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:35.805371    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerID="a128bcb14c41cc0de2c5b26f69460620ebff9b3c286230a9a552833844bd6c0e" exitCode=6
Jul 17 22:22:35.805859 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:35.805430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerDied","Data":"a128bcb14c41cc0de2c5b26f69460620ebff9b3c286230a9a552833844bd6c0e"}
Jul 17 22:22:35.805859 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:35.805672    2563 scope.go:117] "RemoveContainer" containerID="a128bcb14c41cc0de2c5b26f69460620ebff9b3c286230a9a552833844bd6c0e"
Jul 17 22:22:36.812429 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:36.812389    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerID="0760b7e54f13812bbbed36acead4bb3e7410d2c37d5ae085da90638c486efae3" exitCode=6
Jul 17 22:22:36.812913 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:36.812472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerDied","Data":"0760b7e54f13812bbbed36acead4bb3e7410d2c37d5ae085da90638c486efae3"}
Jul 17 22:22:36.812913 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:36.812540    2563 scope.go:117] "RemoveContainer" containerID="a128bcb14c41cc0de2c5b26f69460620ebff9b3c286230a9a552833844bd6c0e"
Jul 17 22:22:36.812913 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:36.812768    2563 scope.go:117] "RemoveContainer" containerID="0760b7e54f13812bbbed36acead4bb3e7410d2c37d5ae085da90638c486efae3"
Jul 17 22:22:36.813044 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:36.812999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb_odh-ai-gateway-infra(0d5f2521-0a2d-4b55-a8f8-b342d8dd7491)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491"
Jul 17 22:22:37.597438 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:37.597408    2563 scope.go:117] "RemoveContainer" containerID="02b01d64ce8289b21880b1e61810d3a28cecc865e3d761c986e92e773d5408e6"
Jul 17 22:22:37.819366 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:37.819337    2563 scope.go:117] "RemoveContainer" containerID="0760b7e54f13812bbbed36acead4bb3e7410d2c37d5ae085da90638c486efae3"
Jul 17 22:22:37.819731 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:37.819532    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb_odh-ai-gateway-infra(0d5f2521-0a2d-4b55-a8f8-b342d8dd7491)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491"
Jul 17 22:22:38.825932 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:38.825898    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4990d03-55db-4e94-94d2-036c98228da6" containerID="749a3f8a48919daf33edfde1ae8c735c8e7a855d65824b42c6d968e8f2ca1cf5" exitCode=6
Jul 17 22:22:38.826341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:38.825980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" event={"ID":"b4990d03-55db-4e94-94d2-036c98228da6","Type":"ContainerDied","Data":"749a3f8a48919daf33edfde1ae8c735c8e7a855d65824b42c6d968e8f2ca1cf5"}
Jul 17 22:22:38.826341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:38.826048    2563 scope.go:117] "RemoveContainer" containerID="02b01d64ce8289b21880b1e61810d3a28cecc865e3d761c986e92e773d5408e6"
Jul 17 22:22:38.826341 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:38.826306    2563 scope.go:117] "RemoveContainer" containerID="749a3f8a48919daf33edfde1ae8c735c8e7a855d65824b42c6d968e8f2ca1cf5"
Jul 17 22:22:38.826654 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:38.826585    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w_odh-ai-gateway-infra(b4990d03-55db-4e94-94d2-036c98228da6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" podUID="b4990d03-55db-4e94-94d2-036c98228da6"
Jul 17 22:22:39.852182 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:39.852149    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"]
Jul 17 22:22:39.981088 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:39.981066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:40.058607 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.058575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4990d03-55db-4e94-94d2-036c98228da6-maas-api-service-ca\") pod \"b4990d03-55db-4e94-94d2-036c98228da6\" (UID: \"b4990d03-55db-4e94-94d2-036c98228da6\") "
Jul 17 22:22:40.058928 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.058906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4990d03-55db-4e94-94d2-036c98228da6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4990d03-55db-4e94-94d2-036c98228da6" (UID: "b4990d03-55db-4e94-94d2-036c98228da6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:40.160225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.160142    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4990d03-55db-4e94-94d2-036c98228da6-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:22:40.839112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.839078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w" event={"ID":"b4990d03-55db-4e94-94d2-036c98228da6","Type":"ContainerDied","Data":"4e60cf3c76a48db1c04a568d2d9549849bac0c76ae2602aab5db98cba888492e"}
Jul 17 22:22:40.839112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.839109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"
Jul 17 22:22:40.839362 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.839136    2563 scope.go:117] "RemoveContainer" containerID="749a3f8a48919daf33edfde1ae8c735c8e7a855d65824b42c6d968e8f2ca1cf5"
Jul 17 22:22:40.856067 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.856040    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"]
Jul 17 22:22:40.858114 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:40.858093    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-fl44w"]
Jul 17 22:22:42.602623 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:42.602572    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4990d03-55db-4e94-94d2-036c98228da6" path="/var/lib/kubelet/pods/b4990d03-55db-4e94-94d2-036c98228da6/volumes"
Jul 17 22:22:49.596854 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:49.596820    2563 scope.go:117] "RemoveContainer" containerID="0760b7e54f13812bbbed36acead4bb3e7410d2c37d5ae085da90638c486efae3"
Jul 17 22:22:49.835190 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:49.835138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"]
Jul 17 22:22:49.888507 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:49.888476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerStarted","Data":"e3bfaeb751c87a7c56b2a8e166092029730538ea58af7bdfbc7d8d7f6b393952"}
Jul 17 22:22:50.897699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:50.897656    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerID="e3bfaeb751c87a7c56b2a8e166092029730538ea58af7bdfbc7d8d7f6b393952" exitCode=6
Jul 17 22:22:50.898099 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:50.897722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerDied","Data":"e3bfaeb751c87a7c56b2a8e166092029730538ea58af7bdfbc7d8d7f6b393952"}
Jul 17 22:22:50.898099 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:50.897806    2563 scope.go:117] "RemoveContainer" containerID="0760b7e54f13812bbbed36acead4bb3e7410d2c37d5ae085da90638c486efae3"
Jul 17 22:22:50.898099 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:50.898089    2563 scope.go:117] "RemoveContainer" containerID="e3bfaeb751c87a7c56b2a8e166092029730538ea58af7bdfbc7d8d7f6b393952"
Jul 17 22:22:50.898359 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:50.898338    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb_odh-ai-gateway-infra(0d5f2521-0a2d-4b55-a8f8-b342d8dd7491)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491"
Jul 17 22:22:50.912726 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:50.912703    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"]
Jul 17 22:22:50.980504 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:50.980467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"]
Jul 17 22:22:52.048265 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.048231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:52.167936 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.167910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491-maas-api-service-ca\") pod \"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491\" (UID: \"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491\") "
Jul 17 22:22:52.168268 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.168242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" (UID: "0d5f2521-0a2d-4b55-a8f8-b342d8dd7491"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:52.268860 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.268828    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:22:52.911219 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.911153    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"
Jul 17 22:22:52.911219 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.911158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb" event={"ID":"0d5f2521-0a2d-4b55-a8f8-b342d8dd7491","Type":"ContainerDied","Data":"cf9e0d19f1773724b37cfe67d4d0f05ef43c09eb72850038cda836ff72d878ea"}
Jul 17 22:22:52.911219 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.911213    2563 scope.go:117] "RemoveContainer" containerID="e3bfaeb751c87a7c56b2a8e166092029730538ea58af7bdfbc7d8d7f6b393952"
Jul 17 22:22:52.927646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.927620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"]
Jul 17 22:22:52.933385 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:52.933364    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-pqpdb"]
Jul 17 22:22:54.602303 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:54.602266    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" path="/var/lib/kubelet/pods/0d5f2521-0a2d-4b55-a8f8-b342d8dd7491/volumes"
Jul 17 22:22:56.271867 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.271829    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"]
Jul 17 22:22:56.272399 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272378    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272403    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272419    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272441 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272442    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272458    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272482    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272487    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272492    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272503    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:56.272537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fc1a1-7006-491d-bc54-49cd150f741d" containerName="revoke-keys"
Jul 17 22:22:56.272869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272578    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272586    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272610    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:22:56.272869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272619    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:22:56.272869 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.272628    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="912b9eb0-f756-42d2-bb15-a02f0d4f25bc" containerName="revoke-keys"
Jul 17 22:22:56.277110 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.277092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:22:56.279534 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.279513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:22:56.282821 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.282800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"]
Jul 17 22:22:56.406831 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.406789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db1062bd-5d54-4057-ac22-1d74d19c4109-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7\" (UID: \"db1062bd-5d54-4057-ac22-1d74d19c4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:22:56.508219 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.508181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db1062bd-5d54-4057-ac22-1d74d19c4109-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7\" (UID: \"db1062bd-5d54-4057-ac22-1d74d19c4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:22:56.508864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.508843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db1062bd-5d54-4057-ac22-1d74d19c4109-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7\" (UID: \"db1062bd-5d54-4057-ac22-1d74d19c4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:22:56.589326 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.589235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:22:56.732689 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.732661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"]
Jul 17 22:22:56.734381 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:22:56.734358    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1062bd_5d54_4057_ac22_1d74d19c4109.slice/crio-eb759904ef70282f2b68db577c7f9ff5d42938300cb68591ad795c4760c6b2b7 WatchSource:0}: Error finding container eb759904ef70282f2b68db577c7f9ff5d42938300cb68591ad795c4760c6b2b7: Status 404 returned error can't find the container with id eb759904ef70282f2b68db577c7f9ff5d42938300cb68591ad795c4760c6b2b7
Jul 17 22:22:56.936139 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:56.936102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" event={"ID":"db1062bd-5d54-4057-ac22-1d74d19c4109","Type":"ContainerStarted","Data":"eb759904ef70282f2b68db577c7f9ff5d42938300cb68591ad795c4760c6b2b7"}
Jul 17 22:22:57.943287 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:57.943251    2563 generic.go:356] "Generic (PLEG): container finished" podID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerID="de40ec5476d5eb607e312860a6957f1f98a27b8fb1783c5af2c505d4dfd2572a" exitCode=6
Jul 17 22:22:57.943698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:57.943299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" event={"ID":"db1062bd-5d54-4057-ac22-1d74d19c4109","Type":"ContainerDied","Data":"de40ec5476d5eb607e312860a6957f1f98a27b8fb1783c5af2c505d4dfd2572a"}
Jul 17 22:22:57.943698 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:57.943471    2563 scope.go:117] "RemoveContainer" containerID="de40ec5476d5eb607e312860a6957f1f98a27b8fb1783c5af2c505d4dfd2572a"
Jul 17 22:22:58.949782 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:58.949746    2563 generic.go:356] "Generic (PLEG): container finished" podID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerID="47dc9d22171cd53b349e8ed2a1b71a8d9557c06b7ffd89884aca5a1337586395" exitCode=6
Jul 17 22:22:58.950194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:58.949830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" event={"ID":"db1062bd-5d54-4057-ac22-1d74d19c4109","Type":"ContainerDied","Data":"47dc9d22171cd53b349e8ed2a1b71a8d9557c06b7ffd89884aca5a1337586395"}
Jul 17 22:22:58.950194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:58.949893    2563 scope.go:117] "RemoveContainer" containerID="de40ec5476d5eb607e312860a6957f1f98a27b8fb1783c5af2c505d4dfd2572a"
Jul 17 22:22:58.950194 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:58.950054    2563 scope.go:117] "RemoveContainer" containerID="47dc9d22171cd53b349e8ed2a1b71a8d9557c06b7ffd89884aca5a1337586395"
Jul 17 22:22:58.950330 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:58.950299    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7_odh-ai-gateway-infra(db1062bd-5d54-4057-ac22-1d74d19c4109)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109"
Jul 17 22:22:59.956354 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:22:59.956323    2563 scope.go:117] "RemoveContainer" containerID="47dc9d22171cd53b349e8ed2a1b71a8d9557c06b7ffd89884aca5a1337586395"
Jul 17 22:22:59.956755 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:22:59.956523    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7_odh-ai-gateway-infra(db1062bd-5d54-4057-ac22-1d74d19c4109)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109"
Jul 17 22:23:04.856291 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:04.856242    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w" podUID="2ffb609c-52d2-4c50-91ee-02be94efb0d6" containerName="main" containerID="cri-o://9f853dfa6aef6996d4937c45d761b3784e693c022dc2bc4b5751dc279b4e98e0" gracePeriod=45
Jul 17 22:23:04.984250 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:04.984208    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ffb609c-52d2-4c50-91ee-02be94efb0d6" containerID="9f853dfa6aef6996d4937c45d761b3784e693c022dc2bc4b5751dc279b4e98e0" exitCode=0
Jul 17 22:23:04.984411 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:04.984286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w" event={"ID":"2ffb609c-52d2-4c50-91ee-02be94efb0d6","Type":"ContainerDied","Data":"9f853dfa6aef6996d4937c45d761b3784e693c022dc2bc4b5751dc279b4e98e0"}
Jul 17 22:23:05.117468 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.117408    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:23:05.294610 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294570    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-home\") pod \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") "
Jul 17 22:23:05.294781 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294679    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tls-certs\") pod \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") "
Jul 17 22:23:05.294781 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g84d5\" (UniqueName: \"kubernetes.io/projected/2ffb609c-52d2-4c50-91ee-02be94efb0d6-kube-api-access-g84d5\") pod \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") "
Jul 17 22:23:05.294781 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tmp-dir\") pod \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") "
Jul 17 22:23:05.294781 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294770    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-dshm\") pod \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") "
Jul 17 22:23:05.294985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294818    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-model-cache\") pod \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\" (UID: \"2ffb609c-52d2-4c50-91ee-02be94efb0d6\") "
Jul 17 22:23:05.294985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.294885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-home" (OuterVolumeSpecName: "home") pod "2ffb609c-52d2-4c50-91ee-02be94efb0d6" (UID: "2ffb609c-52d2-4c50-91ee-02be94efb0d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:05.295099 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.295023    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ffb609c-52d2-4c50-91ee-02be94efb0d6" (UID: "2ffb609c-52d2-4c50-91ee-02be94efb0d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:05.295170 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.295158    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:05.295249 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.295183    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:05.295249 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.295155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ffb609c-52d2-4c50-91ee-02be94efb0d6" (UID: "2ffb609c-52d2-4c50-91ee-02be94efb0d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:05.297075 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.297045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-dshm" (OuterVolumeSpecName: "dshm") pod "2ffb609c-52d2-4c50-91ee-02be94efb0d6" (UID: "2ffb609c-52d2-4c50-91ee-02be94efb0d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:05.297148 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.297074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ffb609c-52d2-4c50-91ee-02be94efb0d6" (UID: "2ffb609c-52d2-4c50-91ee-02be94efb0d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:23:05.297185 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.297146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffb609c-52d2-4c50-91ee-02be94efb0d6-kube-api-access-g84d5" (OuterVolumeSpecName: "kube-api-access-g84d5") pod "2ffb609c-52d2-4c50-91ee-02be94efb0d6" (UID: "2ffb609c-52d2-4c50-91ee-02be94efb0d6"). InnerVolumeSpecName "kube-api-access-g84d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:23:05.395888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.395800    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffb609c-52d2-4c50-91ee-02be94efb0d6-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:05.395888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.395832    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84d5\" (UniqueName: \"kubernetes.io/projected/2ffb609c-52d2-4c50-91ee-02be94efb0d6-kube-api-access-g84d5\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:05.395888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.395842    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:05.395888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.395851    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffb609c-52d2-4c50-91ee-02be94efb0d6-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:05.991821 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.991783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w" event={"ID":"2ffb609c-52d2-4c50-91ee-02be94efb0d6","Type":"ContainerDied","Data":"568fbe425b1eb5723829e7a19d42554e6abd4ffc275d3c9ef59816cc825f3bff"}
Jul 17 22:23:05.991821 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.991803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"
Jul 17 22:23:05.992327 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:05.991835    2563 scope.go:117] "RemoveContainer" containerID="9f853dfa6aef6996d4937c45d761b3784e693c022dc2bc4b5751dc279b4e98e0"
Jul 17 22:23:06.001555 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.001518    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk" podUID="aded8163-d299-4859-9d00-8a48abe4c3a6" containerName="main" containerID="cri-o://9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02" gracePeriod=45
Jul 17 22:23:06.016002 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.015978    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"]
Jul 17 22:23:06.019426 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.019402    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8e6a3a/rate-test-model-8e6a3a-kserve-b48757cff-x229w"]
Jul 17 22:23:06.257315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.257288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:23:06.403765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.403734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-home\") pod \"aded8163-d299-4859-9d00-8a48abe4c3a6\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") "
Jul 17 22:23:06.403765 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.403768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzbgr\" (UniqueName: \"kubernetes.io/projected/aded8163-d299-4859-9d00-8a48abe4c3a6-kube-api-access-nzbgr\") pod \"aded8163-d299-4859-9d00-8a48abe4c3a6\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") "
Jul 17 22:23:06.404027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.403798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aded8163-d299-4859-9d00-8a48abe4c3a6-tls-certs\") pod \"aded8163-d299-4859-9d00-8a48abe4c3a6\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") "
Jul 17 22:23:06.404027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.403822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-tmp-dir\") pod \"aded8163-d299-4859-9d00-8a48abe4c3a6\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") "
Jul 17 22:23:06.404027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.403875    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-dshm\") pod \"aded8163-d299-4859-9d00-8a48abe4c3a6\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") "
Jul 17 22:23:06.404027 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.403933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-model-cache\") pod \"aded8163-d299-4859-9d00-8a48abe4c3a6\" (UID: \"aded8163-d299-4859-9d00-8a48abe4c3a6\") "
Jul 17 22:23:06.404222 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.404076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aded8163-d299-4859-9d00-8a48abe4c3a6" (UID: "aded8163-d299-4859-9d00-8a48abe4c3a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:06.404222 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.404094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-home" (OuterVolumeSpecName: "home") pod "aded8163-d299-4859-9d00-8a48abe4c3a6" (UID: "aded8163-d299-4859-9d00-8a48abe4c3a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:06.404222 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.404202    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-home\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:06.404335 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.404225    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-tmp-dir\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:06.404335 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.404263    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "aded8163-d299-4859-9d00-8a48abe4c3a6" (UID: "aded8163-d299-4859-9d00-8a48abe4c3a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:06.406163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.406142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-dshm" (OuterVolumeSpecName: "dshm") pod "aded8163-d299-4859-9d00-8a48abe4c3a6" (UID: "aded8163-d299-4859-9d00-8a48abe4c3a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:06.406261 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.406239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aded8163-d299-4859-9d00-8a48abe4c3a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aded8163-d299-4859-9d00-8a48abe4c3a6" (UID: "aded8163-d299-4859-9d00-8a48abe4c3a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:23:06.406301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.406266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aded8163-d299-4859-9d00-8a48abe4c3a6-kube-api-access-nzbgr" (OuterVolumeSpecName: "kube-api-access-nzbgr") pod "aded8163-d299-4859-9d00-8a48abe4c3a6" (UID: "aded8163-d299-4859-9d00-8a48abe4c3a6"). InnerVolumeSpecName "kube-api-access-nzbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:23:06.504951 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.504872    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-dshm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:06.504951 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.504899    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aded8163-d299-4859-9d00-8a48abe4c3a6-model-cache\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:06.504951 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.504910    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbgr\" (UniqueName: \"kubernetes.io/projected/aded8163-d299-4859-9d00-8a48abe4c3a6-kube-api-access-nzbgr\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:06.504951 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.504919    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aded8163-d299-4859-9d00-8a48abe4c3a6-tls-certs\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:06.603154 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.603123    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffb609c-52d2-4c50-91ee-02be94efb0d6" path="/var/lib/kubelet/pods/2ffb609c-52d2-4c50-91ee-02be94efb0d6/volumes"
Jul 17 22:23:06.998723 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.998687    2563 generic.go:356] "Generic (PLEG): container finished" podID="aded8163-d299-4859-9d00-8a48abe4c3a6" containerID="9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02" exitCode=0
Jul 17 22:23:06.998723 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.998728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"
Jul 17 22:23:06.999296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.998770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk" event={"ID":"aded8163-d299-4859-9d00-8a48abe4c3a6","Type":"ContainerDied","Data":"9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02"}
Jul 17 22:23:06.999296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.998807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk" event={"ID":"aded8163-d299-4859-9d00-8a48abe4c3a6","Type":"ContainerDied","Data":"ce1657007f14b859d53ac52b3c8b4ac19d7b505c186aefbc304e2bcb65231b59"}
Jul 17 22:23:06.999296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:06.998830    2563 scope.go:117] "RemoveContainer" containerID="9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02"
Jul 17 22:23:07.007701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:07.007684    2563 scope.go:117] "RemoveContainer" containerID="9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02"
Jul 17 22:23:07.007942 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:07.007921    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02\": container with ID starting with 9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02 not found: ID does not exist" containerID="9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02"
Jul 17 22:23:07.008023 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:07.007952    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02"} err="failed to get container status \"9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02\": rpc error: code = NotFound desc = could not find container \"9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02\": container with ID starting with 9db4212b5f6f28e3fa1a24a266fe5323b261a0bc32d7abe99cd499c9c7ec5b02 not found: ID does not exist"
Jul 17 22:23:07.014797 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:07.014773    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"]
Jul 17 22:23:07.016715 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:07.016699    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5401e5/rate-test-model-5401e5-kserve-559c97b57f-c6rnk"]
Jul 17 22:23:08.361670 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.361636    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"]
Jul 17 22:23:08.362061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362023    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffb609c-52d2-4c50-91ee-02be94efb0d6" containerName="main"
Jul 17 22:23:08.362061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffb609c-52d2-4c50-91ee-02be94efb0d6" containerName="main"
Jul 17 22:23:08.362061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aded8163-d299-4859-9d00-8a48abe4c3a6" containerName="main"
Jul 17 22:23:08.362061 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aded8163-d299-4859-9d00-8a48abe4c3a6" containerName="main"
Jul 17 22:23:08.362200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362114    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4990d03-55db-4e94-94d2-036c98228da6" containerName="revoke-keys"
Jul 17 22:23:08.362200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362121    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffb609c-52d2-4c50-91ee-02be94efb0d6" containerName="main"
Jul 17 22:23:08.362200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362127    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5f2521-0a2d-4b55-a8f8-b342d8dd7491" containerName="revoke-keys"
Jul 17 22:23:08.362200 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.362134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aded8163-d299-4859-9d00-8a48abe4c3a6" containerName="main"
Jul 17 22:23:08.366378 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.366357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:08.372210 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.372183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"]
Jul 17 22:23:08.521728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.521693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db0ec9-c196-4789-af73-aff4973d766a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs\" (UID: \"a7db0ec9-c196-4789-af73-aff4973d766a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:08.602733 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.602698    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aded8163-d299-4859-9d00-8a48abe4c3a6" path="/var/lib/kubelet/pods/aded8163-d299-4859-9d00-8a48abe4c3a6/volumes"
Jul 17 22:23:08.623037 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.622972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db0ec9-c196-4789-af73-aff4973d766a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs\" (UID: \"a7db0ec9-c196-4789-af73-aff4973d766a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:08.623545 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.623526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db0ec9-c196-4789-af73-aff4973d766a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs\" (UID: \"a7db0ec9-c196-4789-af73-aff4973d766a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:08.678538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.678510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:08.815268 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:08.815245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"]
Jul 17 22:23:08.817276 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:23:08.817244    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7db0ec9_c196_4789_af73_aff4973d766a.slice/crio-5aa72c95d0b002a130a266ce47f2acb6e12247bf89ccc721b439679fd983f1e3 WatchSource:0}: Error finding container 5aa72c95d0b002a130a266ce47f2acb6e12247bf89ccc721b439679fd983f1e3: Status 404 returned error can't find the container with id 5aa72c95d0b002a130a266ce47f2acb6e12247bf89ccc721b439679fd983f1e3
Jul 17 22:23:09.011315 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:09.011281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerStarted","Data":"3083dab144f80e2021f6041a029e3ba553865a41707d700f1e5d8a30b54495ce"}
Jul 17 22:23:09.011450 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:09.011347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerStarted","Data":"5aa72c95d0b002a130a266ce47f2acb6e12247bf89ccc721b439679fd983f1e3"}
Jul 17 22:23:09.027688 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:09.027642    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" podStartSLOduration=1.027626586 podStartE2EDuration="1.027626586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:09.025279159 +0000 UTC m=+3352.937519447" watchObservedRunningTime="2026-07-17 22:23:09.027626586 +0000 UTC m=+3352.939866875"
Jul 17 22:23:10.017915 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.017881    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7db0ec9-c196-4789-af73-aff4973d766a" containerID="3083dab144f80e2021f6041a029e3ba553865a41707d700f1e5d8a30b54495ce" exitCode=6
Jul 17 22:23:10.018308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.017980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerDied","Data":"3083dab144f80e2021f6041a029e3ba553865a41707d700f1e5d8a30b54495ce"}
Jul 17 22:23:10.018308 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.018181    2563 scope.go:117] "RemoveContainer" containerID="3083dab144f80e2021f6041a029e3ba553865a41707d700f1e5d8a30b54495ce"
Jul 17 22:23:10.165550 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.165515    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"]
Jul 17 22:23:10.170157 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.170140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:10.175840 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.175817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"]
Jul 17 22:23:10.337703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.337669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62524a9e-ff85-4c47-9d8d-25d39ef1a89b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw\" (UID: \"62524a9e-ff85-4c47-9d8d-25d39ef1a89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:10.438293 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.438248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62524a9e-ff85-4c47-9d8d-25d39ef1a89b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw\" (UID: \"62524a9e-ff85-4c47-9d8d-25d39ef1a89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:10.438948 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.438925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62524a9e-ff85-4c47-9d8d-25d39ef1a89b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw\" (UID: \"62524a9e-ff85-4c47-9d8d-25d39ef1a89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:10.481401 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.481368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:10.617053 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:10.617030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"]
Jul 17 22:23:10.619533 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:23:10.619505    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62524a9e_ff85_4c47_9d8d_25d39ef1a89b.slice/crio-e596856fef8ada93d0931165646dd13cf5cfb6245f6f1fe9245834235b80b9fa WatchSource:0}: Error finding container e596856fef8ada93d0931165646dd13cf5cfb6245f6f1fe9245834235b80b9fa: Status 404 returned error can't find the container with id e596856fef8ada93d0931165646dd13cf5cfb6245f6f1fe9245834235b80b9fa
Jul 17 22:23:11.024900 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.024865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw" event={"ID":"62524a9e-ff85-4c47-9d8d-25d39ef1a89b","Type":"ContainerStarted","Data":"75c0cbca5e2c0c141f7b35e764ef7f09a4ebbc8170cef16cb6a54f25b43300c8"}
Jul 17 22:23:11.025357 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.024924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw" event={"ID":"62524a9e-ff85-4c47-9d8d-25d39ef1a89b","Type":"ContainerStarted","Data":"e596856fef8ada93d0931165646dd13cf5cfb6245f6f1fe9245834235b80b9fa"}
Jul 17 22:23:11.026790 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.026765    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7db0ec9-c196-4789-af73-aff4973d766a" containerID="eff4884c9ab69a00d0631de28f54361eade6755a0590df4602db30dc47c5de8f" exitCode=6
Jul 17 22:23:11.026903 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.026840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerDied","Data":"eff4884c9ab69a00d0631de28f54361eade6755a0590df4602db30dc47c5de8f"}
Jul 17 22:23:11.026903 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.026881    2563 scope.go:117] "RemoveContainer" containerID="3083dab144f80e2021f6041a029e3ba553865a41707d700f1e5d8a30b54495ce"
Jul 17 22:23:11.027072 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.027041    2563 scope.go:117] "RemoveContainer" containerID="eff4884c9ab69a00d0631de28f54361eade6755a0590df4602db30dc47c5de8f"
Jul 17 22:23:11.027279 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:11.027260    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs_odh-ai-gateway-infra(a7db0ec9-c196-4789-af73-aff4973d766a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" podUID="a7db0ec9-c196-4789-af73-aff4973d766a"
Jul 17 22:23:11.040958 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.040909    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw" podStartSLOduration=1.040895589 podStartE2EDuration="1.040895589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:11.039625189 +0000 UTC m=+3354.951865476" watchObservedRunningTime="2026-07-17 22:23:11.040895589 +0000 UTC m=+3354.953135878"
Jul 17 22:23:11.597271 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:11.597237    2563 scope.go:117] "RemoveContainer" containerID="47dc9d22171cd53b349e8ed2a1b71a8d9557c06b7ffd89884aca5a1337586395"
Jul 17 22:23:12.033406 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:12.033312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" event={"ID":"db1062bd-5d54-4057-ac22-1d74d19c4109","Type":"ContainerStarted","Data":"78da1bc6f28426bccc963dbff625e65b41d109d6a57d1f537b27687e9aadb1ae"}
Jul 17 22:23:12.034887 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:12.034860    2563 generic.go:356] "Generic (PLEG): container finished" podID="62524a9e-ff85-4c47-9d8d-25d39ef1a89b" containerID="75c0cbca5e2c0c141f7b35e764ef7f09a4ebbc8170cef16cb6a54f25b43300c8" exitCode=0
Jul 17 22:23:12.035033 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:12.034932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw" event={"ID":"62524a9e-ff85-4c47-9d8d-25d39ef1a89b","Type":"ContainerDied","Data":"75c0cbca5e2c0c141f7b35e764ef7f09a4ebbc8170cef16cb6a54f25b43300c8"}
Jul 17 22:23:12.036703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:12.036683    2563 scope.go:117] "RemoveContainer" containerID="eff4884c9ab69a00d0631de28f54361eade6755a0590df4602db30dc47c5de8f"
Jul 17 22:23:12.036928 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:12.036906    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs_odh-ai-gateway-infra(a7db0ec9-c196-4789-af73-aff4973d766a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" podUID="a7db0ec9-c196-4789-af73-aff4973d766a"
Jul 17 22:23:12.049310 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:12.049261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" podStartSLOduration=16.049247215 podStartE2EDuration="16.049247215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:12.047206845 +0000 UTC m=+3355.959447134" watchObservedRunningTime="2026-07-17 22:23:12.049247215 +0000 UTC m=+3355.961487545"
Jul 17 22:23:13.043116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.043085    2563 generic.go:356] "Generic (PLEG): container finished" podID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerID="78da1bc6f28426bccc963dbff625e65b41d109d6a57d1f537b27687e9aadb1ae" exitCode=6
Jul 17 22:23:13.043559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.043158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" event={"ID":"db1062bd-5d54-4057-ac22-1d74d19c4109","Type":"ContainerDied","Data":"78da1bc6f28426bccc963dbff625e65b41d109d6a57d1f537b27687e9aadb1ae"}
Jul 17 22:23:13.043559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.043222    2563 scope.go:117] "RemoveContainer" containerID="47dc9d22171cd53b349e8ed2a1b71a8d9557c06b7ffd89884aca5a1337586395"
Jul 17 22:23:13.044095 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.043558    2563 scope.go:117] "RemoveContainer" containerID="78da1bc6f28426bccc963dbff625e65b41d109d6a57d1f537b27687e9aadb1ae"
Jul 17 22:23:13.044095 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:13.043816    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7_odh-ai-gateway-infra(db1062bd-5d54-4057-ac22-1d74d19c4109)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109"
Jul 17 22:23:13.058163 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.058124    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"]
Jul 17 22:23:13.176675 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.176645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:13.369153 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.369068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62524a9e-ff85-4c47-9d8d-25d39ef1a89b-maas-api-service-ca\") pod \"62524a9e-ff85-4c47-9d8d-25d39ef1a89b\" (UID: \"62524a9e-ff85-4c47-9d8d-25d39ef1a89b\") "
Jul 17 22:23:13.369449 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.369429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62524a9e-ff85-4c47-9d8d-25d39ef1a89b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62524a9e-ff85-4c47-9d8d-25d39ef1a89b" (UID: "62524a9e-ff85-4c47-9d8d-25d39ef1a89b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:13.469715 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:13.469680    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62524a9e-ff85-4c47-9d8d-25d39ef1a89b-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:14.051559 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.051525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw" event={"ID":"62524a9e-ff85-4c47-9d8d-25d39ef1a89b","Type":"ContainerDied","Data":"e596856fef8ada93d0931165646dd13cf5cfb6245f6f1fe9245834235b80b9fa"}
Jul 17 22:23:14.051881 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.051591    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e596856fef8ada93d0931165646dd13cf5cfb6245f6f1fe9245834235b80b9fa"
Jul 17 22:23:14.051881 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.051550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw"
Jul 17 22:23:14.213743 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.213719    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:23:14.289437 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.289402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db1062bd-5d54-4057-ac22-1d74d19c4109-maas-api-service-ca\") pod \"db1062bd-5d54-4057-ac22-1d74d19c4109\" (UID: \"db1062bd-5d54-4057-ac22-1d74d19c4109\") "
Jul 17 22:23:14.289785 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.289763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1062bd-5d54-4057-ac22-1d74d19c4109-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db1062bd-5d54-4057-ac22-1d74d19c4109" (UID: "db1062bd-5d54-4057-ac22-1d74d19c4109"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:14.391039 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:14.390945    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db1062bd-5d54-4057-ac22-1d74d19c4109-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:15.058211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:15.058177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7" event={"ID":"db1062bd-5d54-4057-ac22-1d74d19c4109","Type":"ContainerDied","Data":"eb759904ef70282f2b68db577c7f9ff5d42938300cb68591ad795c4760c6b2b7"}
Jul 17 22:23:15.058211 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:15.058207    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"
Jul 17 22:23:15.058728 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:15.058243    2563 scope.go:117] "RemoveContainer" containerID="78da1bc6f28426bccc963dbff625e65b41d109d6a57d1f537b27687e9aadb1ae"
Jul 17 22:23:15.074066 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:15.074039    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"]
Jul 17 22:23:15.077577 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:15.077549    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-ck2b7"]
Jul 17 22:23:16.603166 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:16.603134    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" path="/var/lib/kubelet/pods/db1062bd-5d54-4057-ac22-1d74d19c4109/volumes"
Jul 17 22:23:20.591669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.591619    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"]
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.591991    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62524a9e-ff85-4c47-9d8d-25d39ef1a89b" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592002    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62524a9e-ff85-4c47-9d8d-25d39ef1a89b" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592101    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592108    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:20.592116 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.592115    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62524a9e-ff85-4c47-9d8d-25d39ef1a89b" containerName="revoke-keys"
Jul 17 22:23:20.596205 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.596184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:20.605202 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.604741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"]
Jul 17 22:23:20.645278 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.645245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03c33b7-f9c7-4841-a91d-c801c1915754-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5\" (UID: \"f03c33b7-f9c7-4841-a91d-c801c1915754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:20.745830 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.745793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03c33b7-f9c7-4841-a91d-c801c1915754-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5\" (UID: \"f03c33b7-f9c7-4841-a91d-c801c1915754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:20.746364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.746346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03c33b7-f9c7-4841-a91d-c801c1915754-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5\" (UID: \"f03c33b7-f9c7-4841-a91d-c801c1915754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:20.909468 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:20.909380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:21.045511 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:21.045485    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"]
Jul 17 22:23:21.047525 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:23:21.047499    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03c33b7_f9c7_4841_a91d_c801c1915754.slice/crio-28e1c66d994781749087eb1f98e25f22906ea5fe3ad8217fd51bb09475e0d27a WatchSource:0}: Error finding container 28e1c66d994781749087eb1f98e25f22906ea5fe3ad8217fd51bb09475e0d27a: Status 404 returned error can't find the container with id 28e1c66d994781749087eb1f98e25f22906ea5fe3ad8217fd51bb09475e0d27a
Jul 17 22:23:21.091787 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:21.091751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" event={"ID":"f03c33b7-f9c7-4841-a91d-c801c1915754","Type":"ContainerStarted","Data":"28e1c66d994781749087eb1f98e25f22906ea5fe3ad8217fd51bb09475e0d27a"}
Jul 17 22:23:22.098888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:22.098794    2563 generic.go:356] "Generic (PLEG): container finished" podID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerID="3b405b0fe8b8cf1fc08ac6d4343e486e6251bfcfd3be156cfa7a361f60301853" exitCode=6
Jul 17 22:23:22.098888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:22.098859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" event={"ID":"f03c33b7-f9c7-4841-a91d-c801c1915754","Type":"ContainerDied","Data":"3b405b0fe8b8cf1fc08ac6d4343e486e6251bfcfd3be156cfa7a361f60301853"}
Jul 17 22:23:22.099286 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:22.099078    2563 scope.go:117] "RemoveContainer" containerID="3b405b0fe8b8cf1fc08ac6d4343e486e6251bfcfd3be156cfa7a361f60301853"
Jul 17 22:23:23.106494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:23.106408    2563 generic.go:356] "Generic (PLEG): container finished" podID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerID="6ca64853df2197cd63d1f2a3e9d4011f9a86a5497be4a7f48731ddb0ae420437" exitCode=6
Jul 17 22:23:23.106494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:23.106476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" event={"ID":"f03c33b7-f9c7-4841-a91d-c801c1915754","Type":"ContainerDied","Data":"6ca64853df2197cd63d1f2a3e9d4011f9a86a5497be4a7f48731ddb0ae420437"}
Jul 17 22:23:23.106991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:23.106527    2563 scope.go:117] "RemoveContainer" containerID="3b405b0fe8b8cf1fc08ac6d4343e486e6251bfcfd3be156cfa7a361f60301853"
Jul 17 22:23:23.106991 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:23.106784    2563 scope.go:117] "RemoveContainer" containerID="6ca64853df2197cd63d1f2a3e9d4011f9a86a5497be4a7f48731ddb0ae420437"
Jul 17 22:23:23.106991 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:23.106980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5_odh-ai-gateway-infra(f03c33b7-f9c7-4841-a91d-c801c1915754)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754"
Jul 17 22:23:24.013682 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.013637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"]
Jul 17 22:23:24.014526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.014494    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:24.014526 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.014527    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:24.014773 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.014757    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1062bd-5d54-4057-ac22-1d74d19c4109" containerName="revoke-keys"
Jul 17 22:23:24.020614 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.020572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:24.021656 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.021629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"]
Jul 17 22:23:24.072554 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.072519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc1432e-cb18-4ad9-819a-611306889dd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm\" (UID: \"4dc1432e-cb18-4ad9-819a-611306889dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:24.113293 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.113271    2563 scope.go:117] "RemoveContainer" containerID="6ca64853df2197cd63d1f2a3e9d4011f9a86a5497be4a7f48731ddb0ae420437"
Jul 17 22:23:24.113669 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:24.113469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5_odh-ai-gateway-infra(f03c33b7-f9c7-4841-a91d-c801c1915754)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754"
Jul 17 22:23:24.173406 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.173372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc1432e-cb18-4ad9-819a-611306889dd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm\" (UID: \"4dc1432e-cb18-4ad9-819a-611306889dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:24.174047 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.174021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc1432e-cb18-4ad9-819a-611306889dd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm\" (UID: \"4dc1432e-cb18-4ad9-819a-611306889dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:24.332776 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.332680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:24.471757 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:24.471725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"]
Jul 17 22:23:24.473027 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:23:24.473000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc1432e_cb18_4ad9_819a_611306889dd6.slice/crio-39f1b1c048547124f5eb3c0fbd3c3005cc486493f42067cec369d4f787e0e893 WatchSource:0}: Error finding container 39f1b1c048547124f5eb3c0fbd3c3005cc486493f42067cec369d4f787e0e893: Status 404 returned error can't find the container with id 39f1b1c048547124f5eb3c0fbd3c3005cc486493f42067cec369d4f787e0e893
Jul 17 22:23:25.119901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:25.119812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" event={"ID":"4dc1432e-cb18-4ad9-819a-611306889dd6","Type":"ContainerStarted","Data":"660feca509c2c8c1b2ce0edc96009bbe7d8ab7dec64a5378417686716709eb2e"}
Jul 17 22:23:25.119901 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:25.119862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" event={"ID":"4dc1432e-cb18-4ad9-819a-611306889dd6","Type":"ContainerStarted","Data":"39f1b1c048547124f5eb3c0fbd3c3005cc486493f42067cec369d4f787e0e893"}
Jul 17 22:23:25.135158 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:25.135110    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" podStartSLOduration=2.135095036 podStartE2EDuration="2.135095036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:25.133859516 +0000 UTC m=+3369.046099805" watchObservedRunningTime="2026-07-17 22:23:25.135095036 +0000 UTC m=+3369.047335325"
Jul 17 22:23:26.127814 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:26.127774    2563 generic.go:356] "Generic (PLEG): container finished" podID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerID="660feca509c2c8c1b2ce0edc96009bbe7d8ab7dec64a5378417686716709eb2e" exitCode=6
Jul 17 22:23:26.128306 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:26.127844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" event={"ID":"4dc1432e-cb18-4ad9-819a-611306889dd6","Type":"ContainerDied","Data":"660feca509c2c8c1b2ce0edc96009bbe7d8ab7dec64a5378417686716709eb2e"}
Jul 17 22:23:26.128306 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:26.128141    2563 scope.go:117] "RemoveContainer" containerID="660feca509c2c8c1b2ce0edc96009bbe7d8ab7dec64a5378417686716709eb2e"
Jul 17 22:23:27.135311 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:27.135228    2563 generic.go:356] "Generic (PLEG): container finished" podID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerID="00aff4c71a160015c61064eac26fdf09d2d4b2d22d8f0ddf7e8652bcca71c1b6" exitCode=6
Jul 17 22:23:27.135747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:27.135317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" event={"ID":"4dc1432e-cb18-4ad9-819a-611306889dd6","Type":"ContainerDied","Data":"00aff4c71a160015c61064eac26fdf09d2d4b2d22d8f0ddf7e8652bcca71c1b6"}
Jul 17 22:23:27.135747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:27.135377    2563 scope.go:117] "RemoveContainer" containerID="660feca509c2c8c1b2ce0edc96009bbe7d8ab7dec64a5378417686716709eb2e"
Jul 17 22:23:27.135747 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:27.135538    2563 scope.go:117] "RemoveContainer" containerID="00aff4c71a160015c61064eac26fdf09d2d4b2d22d8f0ddf7e8652bcca71c1b6"
Jul 17 22:23:27.135852 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:27.135778    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm_odh-ai-gateway-infra(4dc1432e-cb18-4ad9-819a-611306889dd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6"
Jul 17 22:23:27.596550 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:27.596521    2563 scope.go:117] "RemoveContainer" containerID="eff4884c9ab69a00d0631de28f54361eade6755a0590df4602db30dc47c5de8f"
Jul 17 22:23:28.141840 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:28.141796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerStarted","Data":"dd2b4ec1865b4b87fcd35ef4d5013fe34c3fa8f8a41c06ac22772136f60faece"}
Jul 17 22:23:28.143572 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:28.143552    2563 scope.go:117] "RemoveContainer" containerID="00aff4c71a160015c61064eac26fdf09d2d4b2d22d8f0ddf7e8652bcca71c1b6"
Jul 17 22:23:28.143733 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:28.143716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm_odh-ai-gateway-infra(4dc1432e-cb18-4ad9-819a-611306889dd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6"
Jul 17 22:23:29.151558 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:29.151525    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7db0ec9-c196-4789-af73-aff4973d766a" containerID="dd2b4ec1865b4b87fcd35ef4d5013fe34c3fa8f8a41c06ac22772136f60faece" exitCode=6
Jul 17 22:23:29.152043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:29.151623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerDied","Data":"dd2b4ec1865b4b87fcd35ef4d5013fe34c3fa8f8a41c06ac22772136f60faece"}
Jul 17 22:23:29.152043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:29.151687    2563 scope.go:117] "RemoveContainer" containerID="eff4884c9ab69a00d0631de28f54361eade6755a0590df4602db30dc47c5de8f"
Jul 17 22:23:29.152043 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:29.151964    2563 scope.go:117] "RemoveContainer" containerID="dd2b4ec1865b4b87fcd35ef4d5013fe34c3fa8f8a41c06ac22772136f60faece"
Jul 17 22:23:29.152253 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:29.152220    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs_odh-ai-gateway-infra(a7db0ec9-c196-4789-af73-aff4973d766a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" podUID="a7db0ec9-c196-4789-af73-aff4973d766a"
Jul 17 22:23:29.165283 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:29.165263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"]
Jul 17 22:23:30.299825 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:30.299801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:30.430432 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:30.430407    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db0ec9-c196-4789-af73-aff4973d766a-maas-api-service-ca\") pod \"a7db0ec9-c196-4789-af73-aff4973d766a\" (UID: \"a7db0ec9-c196-4789-af73-aff4973d766a\") "
Jul 17 22:23:30.430841 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:30.430819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7db0ec9-c196-4789-af73-aff4973d766a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7db0ec9-c196-4789-af73-aff4973d766a" (UID: "a7db0ec9-c196-4789-af73-aff4973d766a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:30.531882 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:30.531853    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db0ec9-c196-4789-af73-aff4973d766a-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:31.165942 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:31.165912    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"
Jul 17 22:23:31.165942 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:31.165927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs" event={"ID":"a7db0ec9-c196-4789-af73-aff4973d766a","Type":"ContainerDied","Data":"5aa72c95d0b002a130a266ce47f2acb6e12247bf89ccc721b439679fd983f1e3"}
Jul 17 22:23:31.166186 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:31.166001    2563 scope.go:117] "RemoveContainer" containerID="dd2b4ec1865b4b87fcd35ef4d5013fe34c3fa8f8a41c06ac22772136f60faece"
Jul 17 22:23:31.181799 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:31.181777    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"]
Jul 17 22:23:31.187005 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:31.186982    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-c4tcs"]
Jul 17 22:23:32.602977 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:32.602943    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" path="/var/lib/kubelet/pods/a7db0ec9-c196-4789-af73-aff4973d766a/volumes"
Jul 17 22:23:36.370686 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.370652    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"]
Jul 17 22:23:36.371091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.371020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:36.371091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.371031    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:36.371091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.371052    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:36.371091 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.371057    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:36.371259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.371118    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:36.371259 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.371126    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:36.375499 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.375482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:36.380723 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.380689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"]
Jul 17 22:23:36.483112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.483074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dec38ef-fbd0-40eb-839b-c914d325cff1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh\" (UID: \"9dec38ef-fbd0-40eb-839b-c914d325cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:36.583576 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.583545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dec38ef-fbd0-40eb-839b-c914d325cff1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh\" (UID: \"9dec38ef-fbd0-40eb-839b-c914d325cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:36.584184 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.584166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dec38ef-fbd0-40eb-839b-c914d325cff1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh\" (UID: \"9dec38ef-fbd0-40eb-839b-c914d325cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:36.687649 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.687621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:36.823342 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:36.823309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"]
Jul 17 22:23:36.825110 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:23:36.825079    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dec38ef_fbd0_40eb_839b_c914d325cff1.slice/crio-96ee8e972c957634a59184f9750714c14c3c0341c2650f85683b09efed2a44f6 WatchSource:0}: Error finding container 96ee8e972c957634a59184f9750714c14c3c0341c2650f85683b09efed2a44f6: Status 404 returned error can't find the container with id 96ee8e972c957634a59184f9750714c14c3c0341c2650f85683b09efed2a44f6
Jul 17 22:23:37.200509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:37.200466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh" event={"ID":"9dec38ef-fbd0-40eb-839b-c914d325cff1","Type":"ContainerStarted","Data":"66fa0b53f796402382fb6c9368873dd29cfd7638ceb73c33562bae9c806c7db8"}
Jul 17 22:23:37.200692 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:37.200532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh" event={"ID":"9dec38ef-fbd0-40eb-839b-c914d325cff1","Type":"ContainerStarted","Data":"96ee8e972c957634a59184f9750714c14c3c0341c2650f85683b09efed2a44f6"}
Jul 17 22:23:37.216528 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:37.216479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh" podStartSLOduration=1.216465156 podStartE2EDuration="1.216465156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:37.214748637 +0000 UTC m=+3381.126988928" watchObservedRunningTime="2026-07-17 22:23:37.216465156 +0000 UTC m=+3381.128705488"
Jul 17 22:23:37.597140 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:37.597113    2563 scope.go:117] "RemoveContainer" containerID="6ca64853df2197cd63d1f2a3e9d4011f9a86a5497be4a7f48731ddb0ae420437"
Jul 17 22:23:38.207463 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:38.207428    2563 generic.go:356] "Generic (PLEG): container finished" podID="9dec38ef-fbd0-40eb-839b-c914d325cff1" containerID="66fa0b53f796402382fb6c9368873dd29cfd7638ceb73c33562bae9c806c7db8" exitCode=0
Jul 17 22:23:38.207654 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:38.207513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh" event={"ID":"9dec38ef-fbd0-40eb-839b-c914d325cff1","Type":"ContainerDied","Data":"66fa0b53f796402382fb6c9368873dd29cfd7638ceb73c33562bae9c806c7db8"}
Jul 17 22:23:38.209386 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:38.209364    2563 generic.go:356] "Generic (PLEG): container finished" podID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerID="a50dba88be020a5fd9cc080fb8e76ae1e4f82e5e3d2dffd3015929ecca8d6d7d" exitCode=6
Jul 17 22:23:38.209487 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:38.209443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" event={"ID":"f03c33b7-f9c7-4841-a91d-c801c1915754","Type":"ContainerDied","Data":"a50dba88be020a5fd9cc080fb8e76ae1e4f82e5e3d2dffd3015929ecca8d6d7d"}
Jul 17 22:23:38.209535 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:38.209495    2563 scope.go:117] "RemoveContainer" containerID="6ca64853df2197cd63d1f2a3e9d4011f9a86a5497be4a7f48731ddb0ae420437"
Jul 17 22:23:38.209818 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:38.209802    2563 scope.go:117] "RemoveContainer" containerID="a50dba88be020a5fd9cc080fb8e76ae1e4f82e5e3d2dffd3015929ecca8d6d7d"
Jul 17 22:23:38.210054 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:38.210032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5_odh-ai-gateway-infra(f03c33b7-f9c7-4841-a91d-c801c1915754)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754"
Jul 17 22:23:39.248100 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.248064    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"]
Jul 17 22:23:39.358817 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.358781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:39.415764 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.415738    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:39.508502 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.508424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03c33b7-f9c7-4841-a91d-c801c1915754-maas-api-service-ca\") pod \"f03c33b7-f9c7-4841-a91d-c801c1915754\" (UID: \"f03c33b7-f9c7-4841-a91d-c801c1915754\") "
Jul 17 22:23:39.508708 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.508518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dec38ef-fbd0-40eb-839b-c914d325cff1-maas-api-service-ca\") pod \"9dec38ef-fbd0-40eb-839b-c914d325cff1\" (UID: \"9dec38ef-fbd0-40eb-839b-c914d325cff1\") "
Jul 17 22:23:39.508824 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.508757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03c33b7-f9c7-4841-a91d-c801c1915754-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f03c33b7-f9c7-4841-a91d-c801c1915754" (UID: "f03c33b7-f9c7-4841-a91d-c801c1915754"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:39.508939 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.508916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dec38ef-fbd0-40eb-839b-c914d325cff1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9dec38ef-fbd0-40eb-839b-c914d325cff1" (UID: "9dec38ef-fbd0-40eb-839b-c914d325cff1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:39.596470 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.596440    2563 scope.go:117] "RemoveContainer" containerID="00aff4c71a160015c61064eac26fdf09d2d4b2d22d8f0ddf7e8652bcca71c1b6"
Jul 17 22:23:39.609309 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.609291    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dec38ef-fbd0-40eb-839b-c914d325cff1-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:39.609364 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:39.609314    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03c33b7-f9c7-4841-a91d-c801c1915754-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:40.222696 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.222664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh"
Jul 17 22:23:40.222696 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.222684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh" event={"ID":"9dec38ef-fbd0-40eb-839b-c914d325cff1","Type":"ContainerDied","Data":"96ee8e972c957634a59184f9750714c14c3c0341c2650f85683b09efed2a44f6"}
Jul 17 22:23:40.222916 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.222730    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ee8e972c957634a59184f9750714c14c3c0341c2650f85683b09efed2a44f6"
Jul 17 22:23:40.224004 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.223985    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"
Jul 17 22:23:40.224130 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.224003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5" event={"ID":"f03c33b7-f9c7-4841-a91d-c801c1915754","Type":"ContainerDied","Data":"28e1c66d994781749087eb1f98e25f22906ea5fe3ad8217fd51bb09475e0d27a"}
Jul 17 22:23:40.224130 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.224056    2563 scope.go:117] "RemoveContainer" containerID="a50dba88be020a5fd9cc080fb8e76ae1e4f82e5e3d2dffd3015929ecca8d6d7d"
Jul 17 22:23:40.226001 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.225963    2563 generic.go:356] "Generic (PLEG): container finished" podID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerID="34868bda64711efb3a840fe68627fe88df8a11ab55b39527004dfb8c84f6f4e0" exitCode=6
Jul 17 22:23:40.226102 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.226002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" event={"ID":"4dc1432e-cb18-4ad9-819a-611306889dd6","Type":"ContainerDied","Data":"34868bda64711efb3a840fe68627fe88df8a11ab55b39527004dfb8c84f6f4e0"}
Jul 17 22:23:40.226294 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.226279    2563 scope.go:117] "RemoveContainer" containerID="34868bda64711efb3a840fe68627fe88df8a11ab55b39527004dfb8c84f6f4e0"
Jul 17 22:23:40.226479 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:40.226458    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm_odh-ai-gateway-infra(4dc1432e-cb18-4ad9-819a-611306889dd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6"
Jul 17 22:23:40.233638 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.233620    2563 scope.go:117] "RemoveContainer" containerID="00aff4c71a160015c61064eac26fdf09d2d4b2d22d8f0ddf7e8652bcca71c1b6"
Jul 17 22:23:40.254673 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.254647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"]
Jul 17 22:23:40.260085 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.260064    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-2gzs5"]
Jul 17 22:23:40.602383 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:40.602352    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" path="/var/lib/kubelet/pods/f03c33b7-f9c7-4841-a91d-c801c1915754/volumes"
Jul 17 22:23:41.251563 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:41.251529    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"]
Jul 17 22:23:41.399658 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:41.399634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:41.526912 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:41.526826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc1432e-cb18-4ad9-819a-611306889dd6-maas-api-service-ca\") pod \"4dc1432e-cb18-4ad9-819a-611306889dd6\" (UID: \"4dc1432e-cb18-4ad9-819a-611306889dd6\") "
Jul 17 22:23:41.527201 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:41.527178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc1432e-cb18-4ad9-819a-611306889dd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4dc1432e-cb18-4ad9-819a-611306889dd6" (UID: "4dc1432e-cb18-4ad9-819a-611306889dd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:41.627588 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:41.627557    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc1432e-cb18-4ad9-819a-611306889dd6-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:23:42.239743 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:42.239705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm" event={"ID":"4dc1432e-cb18-4ad9-819a-611306889dd6","Type":"ContainerDied","Data":"39f1b1c048547124f5eb3c0fbd3c3005cc486493f42067cec369d4f787e0e893"}
Jul 17 22:23:42.239743 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:42.239727    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"
Jul 17 22:23:42.239999 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:42.239767    2563 scope.go:117] "RemoveContainer" containerID="34868bda64711efb3a840fe68627fe88df8a11ab55b39527004dfb8c84f6f4e0"
Jul 17 22:23:42.263135 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:42.263099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"]
Jul 17 22:23:42.268796 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:42.268773    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-j58nm"]
Jul 17 22:23:42.602209 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:42.602131    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" path="/var/lib/kubelet/pods/4dc1432e-cb18-4ad9-819a-611306889dd6/volumes"
Jul 17 22:23:46.310976 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.310940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"]
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311286    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311296    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311306    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311312    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311321    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311340    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311346    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dec38ef-fbd0-40eb-839b-c914d325cff1" containerName="revoke-keys"
Jul 17 22:23:46.311346 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311351    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dec38ef-fbd0-40eb-839b-c914d325cff1" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311362    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311367    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311420    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311429    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dec38ef-fbd0-40eb-839b-c914d325cff1" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311435    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311441    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311447    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:23:46.311699 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.311458    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7db0ec9-c196-4789-af73-aff4973d766a" containerName="revoke-keys"
Jul 17 22:23:46.315695 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.315678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:23:46.318434 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.318415    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:23:46.323720 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.323370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"]
Jul 17 22:23:46.469160 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.469124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9\" (UID: \"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:23:46.569864 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.569773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9\" (UID: \"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:23:46.570402 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.570379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9\" (UID: \"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:23:46.627537 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.627493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:23:46.764932 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:46.764886    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"]
Jul 17 22:23:46.766985 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:23:46.766956    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb8b5f3_3f28_4c77_bcd8_c6cc0f1497d8.slice/crio-140778440cedab9a36ddc0d5d5a9663773baccae17fa727228cf9d9fd1cf92ca WatchSource:0}: Error finding container 140778440cedab9a36ddc0d5d5a9663773baccae17fa727228cf9d9fd1cf92ca: Status 404 returned error can't find the container with id 140778440cedab9a36ddc0d5d5a9663773baccae17fa727228cf9d9fd1cf92ca
Jul 17 22:23:47.266301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:47.266208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" event={"ID":"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8","Type":"ContainerStarted","Data":"72e626ce9be39ff48e055de2fa3062c02ad56b0cd5614e915b8fc7e05ed0c880"}
Jul 17 22:23:47.266301 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:47.266258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" event={"ID":"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8","Type":"ContainerStarted","Data":"140778440cedab9a36ddc0d5d5a9663773baccae17fa727228cf9d9fd1cf92ca"}
Jul 17 22:23:47.281709 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:47.281654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" podStartSLOduration=1.281639008 podStartE2EDuration="1.281639008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:47.281534564 +0000 UTC m=+3391.193774867" watchObservedRunningTime="2026-07-17 22:23:47.281639008 +0000 UTC m=+3391.193879420"
Jul 17 22:23:48.273204 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:48.273170    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerID="72e626ce9be39ff48e055de2fa3062c02ad56b0cd5614e915b8fc7e05ed0c880" exitCode=6
Jul 17 22:23:48.273204 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:48.273208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" event={"ID":"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8","Type":"ContainerDied","Data":"72e626ce9be39ff48e055de2fa3062c02ad56b0cd5614e915b8fc7e05ed0c880"}
Jul 17 22:23:48.273645 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:48.273466    2563 scope.go:117] "RemoveContainer" containerID="72e626ce9be39ff48e055de2fa3062c02ad56b0cd5614e915b8fc7e05ed0c880"
Jul 17 22:23:49.280589 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:49.280549    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerID="75e5a4e9c2a5efa12a6c8ce75dc052820c0c6af2c4e9639dd6130e8b3532ebd2" exitCode=6
Jul 17 22:23:49.281047 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:49.280623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" event={"ID":"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8","Type":"ContainerDied","Data":"75e5a4e9c2a5efa12a6c8ce75dc052820c0c6af2c4e9639dd6130e8b3532ebd2"}
Jul 17 22:23:49.281047 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:49.280718    2563 scope.go:117] "RemoveContainer" containerID="72e626ce9be39ff48e055de2fa3062c02ad56b0cd5614e915b8fc7e05ed0c880"
Jul 17 22:23:49.281047 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:49.280885    2563 scope.go:117] "RemoveContainer" containerID="75e5a4e9c2a5efa12a6c8ce75dc052820c0c6af2c4e9639dd6130e8b3532ebd2"
Jul 17 22:23:49.281239 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:49.281090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9_odh-ai-gateway-infra(bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8"
Jul 17 22:23:50.289455 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:23:50.289426    2563 scope.go:117] "RemoveContainer" containerID="75e5a4e9c2a5efa12a6c8ce75dc052820c0c6af2c4e9639dd6130e8b3532ebd2"
Jul 17 22:23:50.289871 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:23:50.289658    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9_odh-ai-gateway-infra(bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8"
Jul 17 22:24:01.596985 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:01.596956    2563 scope.go:117] "RemoveContainer" containerID="75e5a4e9c2a5efa12a6c8ce75dc052820c0c6af2c4e9639dd6130e8b3532ebd2"
Jul 17 22:24:02.355836 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:02.355805    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerID="451a21eff463cfc98a91b56701e52f100e277a1a6d07db6c37fb718ac8bd6045" exitCode=6
Jul 17 22:24:02.356003 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:02.355890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" event={"ID":"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8","Type":"ContainerDied","Data":"451a21eff463cfc98a91b56701e52f100e277a1a6d07db6c37fb718ac8bd6045"}
Jul 17 22:24:02.356003 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:02.355961    2563 scope.go:117] "RemoveContainer" containerID="75e5a4e9c2a5efa12a6c8ce75dc052820c0c6af2c4e9639dd6130e8b3532ebd2"
Jul 17 22:24:02.356303 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:02.356281    2563 scope.go:117] "RemoveContainer" containerID="451a21eff463cfc98a91b56701e52f100e277a1a6d07db6c37fb718ac8bd6045"
Jul 17 22:24:02.356554 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:24:02.356524    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9_odh-ai-gateway-infra(bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8"
Jul 17 22:24:03.379883 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:03.379851    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"]
Jul 17 22:24:03.519808 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:03.519779    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:24:03.616800 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:03.616762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8-maas-api-service-ca\") pod \"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8\" (UID: \"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8\") "
Jul 17 22:24:03.617154 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:03.617125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" (UID: "bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:24:03.717372 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:03.717335    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\""
Jul 17 22:24:04.368519 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:04.368486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9" event={"ID":"bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8","Type":"ContainerDied","Data":"140778440cedab9a36ddc0d5d5a9663773baccae17fa727228cf9d9fd1cf92ca"}
Jul 17 22:24:04.368737 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:04.368541    2563 scope.go:117] "RemoveContainer" containerID="451a21eff463cfc98a91b56701e52f100e277a1a6d07db6c37fb718ac8bd6045"
Jul 17 22:24:04.368737 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:04.368502    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"
Jul 17 22:24:04.392569 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:04.392540    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"]
Jul 17 22:24:04.397960 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:04.397934    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wnzg9"]
Jul 17 22:24:04.602215 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:04.602183    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" path="/var/lib/kubelet/pods/bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8/volumes"
Jul 17 22:24:08.977899 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:08.977817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lkgss_9524360e-21e3-4701-bb03-441a60607603/manager/0.log"
Jul 17 22:24:09.585538 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:09.585509    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-lnxjh_64e8a846-c582-4d3d-966c-4ae48195e2cc/copy-manifests/0.log"
Jul 17 22:24:09.593648 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:09.593591    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-lnxjh_64e8a846-c582-4d3d-966c-4ae48195e2cc/manager/0.log"
Jul 17 22:24:10.440214 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:10.440188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738760-v2fmw_e90a159c-47e1-4d36-a9ea-017373f65754/cleanup/0.log"
Jul 17 22:24:10.556816 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:10.556783    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738775-28cjv_8a783210-9c7e-406a-9250-ea67d641029c/cleanup/0.log"
Jul 17 22:24:10.673889 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:10.673859    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-95299c-sd7tf_a406968c-c87e-4a5b-a992-5197e56e1e65/revoke-keys/0.log"
Jul 17 22:24:10.785873 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:10.785793    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-6fbd36-7hhd8_c646af7a-6101-468a-b006-f44bd04a3582/revoke-keys/0.log"
Jul 17 22:24:10.897774 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:10.897741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-be8c97-fkj9c_ba1c78ec-9984-448b-a467-aac0c49317e7/revoke-keys/0.log"
Jul 17 22:24:11.008358 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:11.008324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-b09be9-phwng_f8487662-bff1-4cfe-9f61-551a03d48a3c/revoke-keys/0.log"
Jul 17 22:24:11.227495 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:11.227463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-539e6669-kntrx_bd7042b7-f051-498e-8669-0b79ef23fa30/revoke-keys/0.log"
Jul 17 22:24:11.337621 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:11.337572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-8962d973-7bk9w_471ffa41-95f6-4321-b55f-605d349f71db/revoke-keys/0.log"
Jul 17 22:24:11.445050 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:11.445024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-8e6a3a-wt2pw_62524a9e-ff85-4c47-9d8d-25d39ef1a89b/revoke-keys/0.log"
Jul 17 22:24:11.556543 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:11.556462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-5401e5-wvblh_9dec38ef-fbd0-40eb-839b-c914d325cff1/revoke-keys/0.log"
Jul 17 22:24:11.665124 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:11.665096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-6vgvh_e24a9d21-c7ec-446d-82e6-7aca444279e1/postgres/0.log"
Jul 17 22:24:13.722822 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:13.722788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-p86js_2e01cf0c-c4d9-469e-865c-91fd1f63ed8c/manager/0.log"
Jul 17 22:24:14.659149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:14.659110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-x5xsr_d4e797a3-5d05-4c8b-ad1d-2dad0f014daf/payload-pre-processing/0.log"
Jul 17 22:24:14.784440 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:14.784410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s72bv_aeb09007-48a1-4669-ad10-c05bc2ac93d5/payload-processing/0.log"
Jul 17 22:24:14.903956 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:14.903928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c474549d4-twbtw_817083d7-66c5-4f1b-9f4d-d30a0de2607c/router/0.log"
Jul 17 22:24:15.249095 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:15.249064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7czwx_13a51e4f-ec36-4607-af46-a9059d9928ab/main/0.log"
Jul 17 22:24:15.478585 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:15.478555    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5fv2x_c116926b-b349-4a5e-a311-71a0ff5e27d2/main/0.log"
Jul 17 22:24:24.639010 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:24.638973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dz78x_0a956c0c-f929-4b1c-a9e1-7449f408c8a7/global-pull-secret-syncer/0.log"
Jul 17 22:24:24.838396 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:24.838359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sqnqz_76b0098e-e168-452f-a057-d6488676bb28/konnectivity-agent/0.log"
Jul 17 22:24:24.884497 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:24.884464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-16.ec2.internal_1a28b61f94e609f57bdc8bf0691328b0/haproxy/0.log"
Jul 17 22:24:30.503149 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:30.503115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-p86js_2e01cf0c-c4d9-469e-865c-91fd1f63ed8c/manager/0.log"
Jul 17 22:24:32.255492 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:32.255438    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-72h96_6ee8f435-8f90-403c-b956-2359bbc15013/cluster-monitoring-operator/0.log"
Jul 17 22:24:32.494090 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:32.494004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dfchh_d165ff68-206a-451e-b307-bfcc5c190313/node-exporter/0.log"
Jul 17 22:24:32.513619 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:32.513574    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dfchh_d165ff68-206a-451e-b307-bfcc5c190313/kube-rbac-proxy/0.log"
Jul 17 22:24:32.536643 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:32.536622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dfchh_d165ff68-206a-451e-b307-bfcc5c190313/init-textfile/0.log"
Jul 17 22:24:34.423239 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:34.423207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-5njn5_8d9fef3e-0416-41cd-8c83-831674459a8c/networking-console-plugin/0.log"
Jul 17 22:24:35.014112 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014073    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"]
Jul 17 22:24:35.014457 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014443    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:24:35.014509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:24:35.014509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014482    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:24:35.014509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014488    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc1432e-cb18-4ad9-819a-611306889dd6" containerName="revoke-keys"
Jul 17 22:24:35.014509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014498    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014509 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014503    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014520    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014535    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014609    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014622    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014631    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb8b5f3-3f28-4c77-bcd8-c6cc0f1497d8" containerName="revoke-keys"
Jul 17 22:24:35.014710 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.014638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03c33b7-f9c7-4841-a91d-c801c1915754" containerName="revoke-keys"
Jul 17 22:24:35.017946 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.017926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.020302 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.020285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5qg69\"/\"default-dockercfg-ljcb8\""
Jul 17 22:24:35.020392 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.020290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qg69\"/\"openshift-service-ca.crt\""
Jul 17 22:24:35.021192 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.021178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qg69\"/\"kube-root-ca.crt\""
Jul 17 22:24:35.026129 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.026106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"]
Jul 17 22:24:35.054486 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.054450    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/1.log"
Jul 17 22:24:35.064731 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.064705    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-95s5k_a0dd0ed2-41e2-4790-8122-7ea4cc680c95/console-operator/2.log"
Jul 17 22:24:35.090636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.090574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-lib-modules\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.090636 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.090647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-podres\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.090839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.090666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-sys\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.090839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.090681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzm7\" (UniqueName: \"kubernetes.io/projected/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-kube-api-access-tbzm7\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.090839 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.090751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-proc\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191654 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-proc\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-lib-modules\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-podres\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-proc\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-sys\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzm7\" (UniqueName: \"kubernetes.io/projected/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-kube-api-access-tbzm7\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.191805 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-sys\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.192049 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-podres\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.192049 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.191858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-lib-modules\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.200390 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.200370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzm7\" (UniqueName: \"kubernetes.io/projected/aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd-kube-api-access-tbzm7\") pod \"perf-node-gather-daemonset-6pqqt\" (UID: \"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.329371 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.329291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:35.468423 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.468395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"]
Jul 17 22:24:35.471235 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:24:35.471200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa87f8d2_ffd7_4e4c_9d0e_0b7a1f8f9fbd.slice/crio-84829e5d0af985d2802795e44971034144055fead6d9d403bcc1c36f06ab6c4e WatchSource:0}: Error finding container 84829e5d0af985d2802795e44971034144055fead6d9d403bcc1c36f06ab6c4e: Status 404 returned error can't find the container with id 84829e5d0af985d2802795e44971034144055fead6d9d403bcc1c36f06ab6c4e
Jul 17 22:24:35.528634 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.528585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt" event={"ID":"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd","Type":"ContainerStarted","Data":"84829e5d0af985d2802795e44971034144055fead6d9d403bcc1c36f06ab6c4e"}
Jul 17 22:24:35.672009 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.671986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c765ccb77-5755m_0e0db462-8e7f-4041-bdc7-4037ea7e23c4/console/0.log"
Jul 17 22:24:35.724106 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:35.724071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-4ztx7_c2081059-6078-4527-9218-fe64c24b2624/download-server/0.log"
Jul 17 22:24:36.292578 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.292546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-4nwfb_fbed4061-adc3-4a72-98e1-ed314e75909e/volume-data-source-validator/0.log"
Jul 17 22:24:36.350962 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.350933    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"]
Jul 17 22:24:36.354669 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.354652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"
Jul 17 22:24:36.357133 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.357109    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\""
Jul 17 22:24:36.361955 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.361926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"]
Jul 17 22:24:36.503703 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.503669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0bad2-8d37-4ca6-8395-b75d58f570ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46\" (UID: \"1cc0bad2-8d37-4ca6-8395-b75d58f570ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"
Jul 17 22:24:36.534814 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.534782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt" event={"ID":"aa87f8d2-ffd7-4e4c-9d0e-0b7a1f8f9fbd","Type":"ContainerStarted","Data":"3d9875b4819393c16ae22f20acab75a4578892709e94220d9cde26641b2017b2"}
Jul 17 22:24:36.534968 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.534910    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:36.550693 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.550590    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt" podStartSLOduration=2.550577929 podStartE2EDuration="2.550577929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:36.549198258 +0000 UTC m=+3440.461438558" watchObservedRunningTime="2026-07-17 22:24:36.550577929 +0000 UTC m=+3440.462818217"
Jul 17 22:24:36.604272 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.604239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0bad2-8d37-4ca6-8395-b75d58f570ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46\" (UID: \"1cc0bad2-8d37-4ca6-8395-b75d58f570ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"
Jul 17 22:24:36.604831 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.604812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0bad2-8d37-4ca6-8395-b75d58f570ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46\" (UID: \"1cc0bad2-8d37-4ca6-8395-b75d58f570ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"
Jul 17 22:24:36.667254 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:36.667227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"
Jul 17 22:24:37.013480 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.013445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46"]
Jul 17 22:24:37.014735 ip-10-0-139-16 kubenswrapper[2563]: W0717 22:24:37.014711    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc0bad2_8d37_4ca6_8395_b75d58f570ac.slice/crio-7692900a5b1f6b8f32d09fdc06c6a573db91a67dc02eac9c0266b4185d07a934 WatchSource:0}: Error finding container 7692900a5b1f6b8f32d09fdc06c6a573db91a67dc02eac9c0266b4185d07a934: Status 404 returned error can't find the container with id 7692900a5b1f6b8f32d09fdc06c6a573db91a67dc02eac9c0266b4185d07a934
Jul 17 22:24:37.289417 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.289382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b6cwm_5f9d5559-38df-4b36-90d6-3c572fb461d4/dns/0.log"
Jul 17 22:24:37.310093 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.310066    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b6cwm_5f9d5559-38df-4b36-90d6-3c572fb461d4/kube-rbac-proxy/0.log"
Jul 17 22:24:37.470727 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.470681    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97sf8_a05c180a-04d3-4fa2-904d-034acaf5ada9/dns-node-resolver/0.log"
Jul 17 22:24:37.541029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.540923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" event={"ID":"1cc0bad2-8d37-4ca6-8395-b75d58f570ac","Type":"ContainerStarted","Data":"bf38b38a74529fdac9dca5d8256583816f3e57a2c0cdabeabc8991aabf168d8e"}
Jul 17 22:24:37.541029 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.540989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" event={"ID":"1cc0bad2-8d37-4ca6-8395-b75d58f570ac","Type":"ContainerStarted","Data":"7692900a5b1f6b8f32d09fdc06c6a573db91a67dc02eac9c0266b4185d07a934"}
Jul 17 22:24:37.557435 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:37.557387    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" podStartSLOduration=1.5573693469999998 podStartE2EDuration="1.557369347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:37.555115756 +0000 UTC m=+3441.467356045" watchObservedRunningTime="2026-07-17 22:24:37.557369347 +0000 UTC m=+3441.469609635"
Jul 17 22:24:38.059808 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:38.059767    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9c7k_4fdc8bd8-1135-46ad-a9e9-4e06928fc07e/node-ca/0.log"
Jul 17 22:24:38.547646 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:38.547578    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cc0bad2-8d37-4ca6-8395-b75d58f570ac" containerID="bf38b38a74529fdac9dca5d8256583816f3e57a2c0cdabeabc8991aabf168d8e" exitCode=6
Jul 17 22:24:38.548051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:38.547674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" event={"ID":"1cc0bad2-8d37-4ca6-8395-b75d58f570ac","Type":"ContainerDied","Data":"bf38b38a74529fdac9dca5d8256583816f3e57a2c0cdabeabc8991aabf168d8e"}
Jul 17 22:24:38.548051 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:38.547807    2563 scope.go:117] "RemoveContainer" containerID="bf38b38a74529fdac9dca5d8256583816f3e57a2c0cdabeabc8991aabf168d8e"
Jul 17 22:24:39.271900 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.271866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-x5xsr_d4e797a3-5d05-4c8b-ad1d-2dad0f014daf/payload-pre-processing/0.log"
Jul 17 22:24:39.313307 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.313284    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s72bv_aeb09007-48a1-4669-ad10-c05bc2ac93d5/payload-processing/0.log"
Jul 17 22:24:39.339143 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.339105    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c474549d4-twbtw_817083d7-66c5-4f1b-9f4d-d30a0de2607c/router/0.log"
Jul 17 22:24:39.554494 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.554405    2563 generic.go:356] "Generic (PLEG): container finished" podID="1cc0bad2-8d37-4ca6-8395-b75d58f570ac" containerID="3395b2a4847b5bf7095395b6b69698d75e926c3a071d11d8e72fd40d12ec73a5" exitCode=6
Jul 17 22:24:39.554909 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.554496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" event={"ID":"1cc0bad2-8d37-4ca6-8395-b75d58f570ac","Type":"ContainerDied","Data":"3395b2a4847b5bf7095395b6b69698d75e926c3a071d11d8e72fd40d12ec73a5"}
Jul 17 22:24:39.554909 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.554560    2563 scope.go:117] "RemoveContainer" containerID="bf38b38a74529fdac9dca5d8256583816f3e57a2c0cdabeabc8991aabf168d8e"
Jul 17 22:24:39.554909 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:39.554750    2563 scope.go:117] "RemoveContainer" containerID="3395b2a4847b5bf7095395b6b69698d75e926c3a071d11d8e72fd40d12ec73a5"
Jul 17 22:24:39.555026 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:24:39.554974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46_odh-ai-gateway-infra(1cc0bad2-8d37-4ca6-8395-b75d58f570ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" podUID="1cc0bad2-8d37-4ca6-8395-b75d58f570ac"
Jul 17 22:24:40.085893 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:40.085849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-psfq2_600ae2a5-5709-473f-b2c1-63145e434e91/serve-healthcheck-canary/0.log"
Jul 17 22:24:40.542038 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:40.542008    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-62d22_ea756d28-1f16-4d91-ad54-d047cb94f4b0/insights-operator/0.log"
Jul 17 22:24:40.560888 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:40.560868    2563 scope.go:117] "RemoveContainer" containerID="3395b2a4847b5bf7095395b6b69698d75e926c3a071d11d8e72fd40d12ec73a5"
Jul 17 22:24:40.561300 ip-10-0-139-16 kubenswrapper[2563]: E0717 22:24:40.561045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46_odh-ai-gateway-infra(1cc0bad2-8d37-4ca6-8395-b75d58f570ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-x6b46" podUID="1cc0bad2-8d37-4ca6-8395-b75d58f570ac"
Jul 17 22:24:40.656492 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:40.656459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bk6gm_cf151b43-656f-4727-b3cf-2b97d0c3d045/kube-rbac-proxy/0.log"
Jul 17 22:24:40.704502 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:40.704473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bk6gm_cf151b43-656f-4727-b3cf-2b97d0c3d045/exporter/0.log"
Jul 17 22:24:40.786218 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:40.786192    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bk6gm_cf151b43-656f-4727-b3cf-2b97d0c3d045/extractor/0.log"
Jul 17 22:24:42.552155 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:42.552129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-6pqqt"
Jul 17 22:24:42.930212 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:42.930176    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lkgss_9524360e-21e3-4701-bb03-441a60607603/manager/0.log"
Jul 17 22:24:43.364548 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:43.364460    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-lnxjh_64e8a846-c582-4d3d-966c-4ae48195e2cc/manager/0.log"
Jul 17 22:24:43.382702 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:43.382680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-lnxjh_64e8a846-c582-4d3d-966c-4ae48195e2cc/copy-manifests/0.log"
Jul 17 22:24:44.993183 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:44.993144    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bbc9cc6cd-lkmtb_31a4d9f3-81cd-4c53-a978-82ea947e0f5f/manager/0.log"
Jul 17 22:24:45.756415 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:45.756383    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-x5xsr_d4e797a3-5d05-4c8b-ad1d-2dad0f014daf/payload-pre-processing/0.log"
Jul 17 22:24:45.786551 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:45.786511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s72bv_aeb09007-48a1-4669-ad10-c05bc2ac93d5/payload-processing/0.log"
Jul 17 22:24:45.804701 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:45.804665    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c474549d4-twbtw_817083d7-66c5-4f1b-9f4d-d30a0de2607c/router/0.log"
Jul 17 22:24:47.212791 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:47.212763    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-x5xsr_d4e797a3-5d05-4c8b-ad1d-2dad0f014daf/payload-pre-processing/0.log"
Jul 17 22:24:47.243616 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:47.243552    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s72bv_aeb09007-48a1-4669-ad10-c05bc2ac93d5/payload-processing/0.log"
Jul 17 22:24:47.261296 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:47.261264    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c474549d4-twbtw_817083d7-66c5-4f1b-9f4d-d30a0de2607c/router/0.log"
Jul 17 22:24:49.897304 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:49.897271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-7s68h_ac42b126-9d2b-47fa-8926-fa792552e396/migrator/0.log"
Jul 17 22:24:49.915510 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:49.915485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-7s68h_ac42b126-9d2b-47fa-8926-fa792552e396/graceful-termination/0.log"
Jul 17 22:24:50.253753 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:50.253720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-lsnt6_76d7418b-0187-4848-a3e1-8fcbab28dc4b/kube-storage-version-migrator-operator/1.log"
Jul 17 22:24:50.255202 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:50.255172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-lsnt6_76d7418b-0187-4848-a3e1-8fcbab28dc4b/kube-storage-version-migrator-operator/0.log"
Jul 17 22:24:51.192225 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.192148    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/kube-multus-additional-cni-plugins/0.log"
Jul 17 22:24:51.213079 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.213044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/egress-router-binary-copy/0.log"
Jul 17 22:24:51.231708 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.231686    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/cni-plugins/0.log"
Jul 17 22:24:51.250423 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.250397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/bond-cni-plugin/0.log"
Jul 17 22:24:51.270015 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.269996    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/routeoverride-cni/0.log"
Jul 17 22:24:51.288394 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.288374    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/whereabouts-cni-bincopy/0.log"
Jul 17 22:24:51.307489 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.307470    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27mht_40814c12-245a-4369-bfc8-4ea9cd93e60f/whereabouts-cni/0.log"
Jul 17 22:24:51.709388 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.709347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-swkc4_90e3694e-96d4-4374-b47e-b666fc83db50/kube-multus/0.log"
Jul 17 22:24:51.863060 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.863028    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jqdbf_68e20af5-a9a5-464f-81d1-67a8c17db4cf/network-metrics-daemon/0.log"
Jul 17 22:24:51.877868 ip-10-0-139-16 kubenswrapper[2563]: I0717 22:24:51.877845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jqdbf_68e20af5-a9a5-464f-81d1-67a8c17db4cf/kube-rbac-proxy/0.log"