Jul 17 04:49:56.148555 ip-10-0-140-139 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 04:49:56.537236 ip-10-0-140-139 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 04:49:56.537236 ip-10-0-140-139 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 04:49:56.537236 ip-10-0-140-139 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 04:49:56.537236 ip-10-0-140-139 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 04:49:56.537236 ip-10-0-140-139 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 04:49:56.538118 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.537996    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 04:49:56.543067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543049    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 04:49:56.543067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543067    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543072    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543076    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543079    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543082    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543086    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543089    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543092    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543095    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543098    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543100    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543105    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543109    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543113    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543124    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543127    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543130    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543133    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543136    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 04:49:56.543140 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543139    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543143    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543147    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543150    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543153    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543156    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543159    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543161    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543164    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543166    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543169    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543171    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543174    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543176    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543178    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543181    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543183    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543186    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543188    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543190    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 04:49:56.543608 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543193    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543195    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543198    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543200    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543203    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543205    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543207    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543210    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543213    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543216    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543220    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543222    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543225    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543228    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543231    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543234    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543237    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543239    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543242    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 04:49:56.544079 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543244    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543247    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543250    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543252    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543269    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543272    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543274    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543277    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543280    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543283    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543285    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543288    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543290    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543292    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543296    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543298    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543301    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543303    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543305    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543308    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 04:49:56.544596 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543312    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543315    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543318    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543321    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543323    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543325    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543328    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543331    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543334    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543337    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543339    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543342    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543344    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543347    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543349    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543352    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543354    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543357    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543359    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.543362    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 04:49:56.545067 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544936    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544944    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544948    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544951    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544953    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544956    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544959    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544962    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544965    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544967    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544970    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544973    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544976    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544979    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544982    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544984    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544987    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544989    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544991    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544994    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 04:49:56.545578 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.544997    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545000    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545002    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545005    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545007    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545010    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545012    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545015    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545019    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545022    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545026    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545040    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545044    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545047    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545050    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545053    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545055    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545058    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545061    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 04:49:56.546106 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545064    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545067    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545069    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545071    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545074    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545077    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545080    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545083    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545086    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545088    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545091    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545093    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545097    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545100    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545102    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545105    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545108    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545110    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545113    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545116    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 04:49:56.546592 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545118    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545120    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545123    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545125    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545128    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545130    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545133    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545135    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545138    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545140    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545143    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545145    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545147    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545150    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545152    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545154    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545157    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545159    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545161    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545165    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 04:49:56.547086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545169    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545172    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545175    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545177    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545180    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545183    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545186    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545188    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545191    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545193    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545196    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545198    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545200    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545203    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545206    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545208    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545210    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545213    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545215    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545218    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 04:49:56.547579 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545599    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545603    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545606    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545609    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545611    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545614    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545616    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545619    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545622    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545624    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545627    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545630    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545633    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545635    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545638    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545640    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545643    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545646    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545648    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 04:49:56.548086 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545651    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545654    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545656    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545658    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545661    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545663    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545665    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545668    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545670    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545673    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545675    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545678    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545680    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545682    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545685    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545687    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545689    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545691    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545694    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545696    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 04:49:56.548559 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545699    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545701    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545703    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545707    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545709    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545712    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545715    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545717    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545720    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545723    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545726    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545728    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545730    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545733    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545735    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545738    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545740    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545742    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545745    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545747    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 04:49:56.549037 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545750    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545753    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545755    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545758    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545760    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545762    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545765    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545767    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545769    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545772    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545774    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545777    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545779    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545782    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545784    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545787    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545791    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545795    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545797    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545800    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 04:49:56.549563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545803    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545807    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545812    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545816    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545820    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545824    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545829    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545876    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545883    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545886    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545889    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545892    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545895    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545898    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545900    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545905    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545907    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545912    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545916    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 04:49:56.550058 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.545919    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546018    2582 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546029    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546039    2582 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546045    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546052    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546057    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546064    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546070    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546075    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546080    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546086    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546090    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546095    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546099    2582 flags.go:64] FLAG: --cgroup-root=""
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546104    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546109    2582 flags.go:64] FLAG: --client-ca-file=""
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546114    2582 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546118    2582 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546124    2582 flags.go:64] FLAG: --cluster-domain=""
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546129    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546134    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546139    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546144    2582 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 04:49:56.550542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546150    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546155    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546160    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546166    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546171    2582 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546176    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546180    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546187    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546191    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546198    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546203    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546208    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546213    2582 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546218    2582 flags.go:64] FLAG: --enable-server="true"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546223    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546231    2582 flags.go:64] FLAG: --event-burst="100"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546236    2582 flags.go:64] FLAG: --event-qps="50"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546241    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546246    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546251    2582 flags.go:64] FLAG: --eviction-hard=""
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546272    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546277    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546282    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546286    2582 flags.go:64] FLAG: --eviction-soft=""
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546292    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 04:49:56.551121 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546297    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546302    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546307    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546312    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546317    2582 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546322    2582 flags.go:64] FLAG: --feature-gates=""
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546327    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546332    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546337    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546342    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546346    2582 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546351    2582 flags.go:64] FLAG: --help="false"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546355    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546360    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546365    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546369    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546374    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546379    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546384    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546389    2582 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546392    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546395    2582 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546398    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546401    2582 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 04:49:56.551742 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546404    2582 flags.go:64] FLAG: --kube-reserved=""
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546407    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546410    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546413    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546416    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546419    2582 flags.go:64] FLAG: --lock-file=""
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546421    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546427    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546430    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546435    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546438    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546441    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546444    2582 flags.go:64] FLAG: --logging-format="text"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546447    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546451    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546454    2582 flags.go:64] FLAG: --manifest-url=""
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546457    2582 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546475    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546478    2582 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546482    2582 flags.go:64] FLAG: --max-pods="110"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546485    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546488    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546491    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546494    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546497    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 04:49:56.552345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546500    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546503    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546511    2582 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546515    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546518    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546521    2582 flags.go:64] FLAG: --pod-cidr=""
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546524    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546530    2582 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546533    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546536    2582 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546539    2582 flags.go:64] FLAG: --port="10250"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546542    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546545    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-017fa57dcb8933c45"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546547    2582 flags.go:64] FLAG: --qos-reserved=""
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546550    2582 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546554    2582 flags.go:64] FLAG: --register-node="true"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546557    2582 flags.go:64] FLAG: --register-with-taints=""
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546562    2582 flags.go:64] FLAG: --registry-burst="10"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546565    2582 flags.go:64] FLAG: --registry-qps="5"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546568    2582 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546571    2582 flags.go:64] FLAG: --reserved-memory=""
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546574    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546578    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546581    2582 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546584    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 04:49:56.552972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546587    2582 flags.go:64] FLAG: --runonce="false"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546590    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546593    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546596    2582 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546599    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546602    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546605    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546608    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546611    2582 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546614    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546617    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546620    2582 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546623    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546626    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546629    2582 flags.go:64] FLAG: --system-cgroups=""
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546631    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546637    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546640    2582 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546642    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546647    2582 flags.go:64] FLAG: --tls-min-version=""
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546649    2582 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546652    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546658    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546661    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546665    2582 flags.go:64] FLAG: --v="2"
Jul 17 04:49:56.553631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546669    2582 flags.go:64] FLAG: --version="false"
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546673    2582 flags.go:64] FLAG: --vmodule=""
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546677    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.546680    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546802    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546806    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546809    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546811    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546814    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546816    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546819    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546822    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546824    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546826    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546829    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546832    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546834    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546836    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546839    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546841    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 04:49:56.554237 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546844    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546846    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546848    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546851    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546853    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546856    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546858    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546861    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546863    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546867    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546870    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546874    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546876    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546879    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546881    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546885    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546887    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546889    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546892    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546895    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 04:49:56.554735 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546897    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546899    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546902    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546904    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546907    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546910    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546912    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546915    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546917    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546919    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546922    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546924    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546927    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546929    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546931    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546934    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546937    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546939    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546942    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546944    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 04:49:56.555350 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546947    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546950    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546953    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546957    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546959    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546962    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546964    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546967    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546969    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546971    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546974    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546978    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546980    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546983    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546985    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546987    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546991    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546995    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.546998    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 04:49:56.555842 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547002    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547004    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547009    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547012    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547015    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547017    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547019    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547022    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547024    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547027    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547030    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547032    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547036    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547039    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547043    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547045    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547049    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547052    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547054    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 04:49:56.556325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547057    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547059    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547062    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547065    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.547067    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.547640    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.554635    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.554652    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.554672    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.554679    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554749    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554756    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554760    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554763    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 04:49:56.556782 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554767    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554771    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554774    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554777    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554780    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554782    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554785    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554788    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554790    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554793    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554795    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554798    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554801    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554803    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554806    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554808    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554811    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554813    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554816    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554830    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 04:49:56.557142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554834    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554837    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554840    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554843    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554847    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554850    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554853    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554856    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554858    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554861    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554863    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554866    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554868    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554871    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554873    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554876    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554879    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554881    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554884    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554887    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 04:49:56.557632 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554889    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554892    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554894    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554897    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554900    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554902    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554905    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554908    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554910    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554913    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554915    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554918    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554922    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554925    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554929    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554932    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554935    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554937    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554940    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 04:49:56.558144 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554943    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554945    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554948    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554950    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554952    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554955    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554957    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554960    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554962    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554965    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554967    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554970    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554973    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554975    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554977    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554980    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554983    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554986    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554988    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554990    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 04:49:56.558659 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554993    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554996    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.554998    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555001    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555004    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555006    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555008    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555011    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555013    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555016    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555019    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555021    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555024    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555027    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555029    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 04:49:56.559139 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555031    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.555036    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555151    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555156    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555159    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555162    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555165    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555167    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555170    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555172    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555175    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555177    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555180    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555182    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555185    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 04:49:56.559918 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555187    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555190    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555192    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555195    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555198    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555202    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555205    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555208    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555211    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555213    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555216    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555218    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555221    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555223    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555225    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555228    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555231    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555235    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555238    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 04:49:56.560527 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555241    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555244    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555246    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555249    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555251    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555254    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555275    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555279    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555282    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555284    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555287    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555289    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555292    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555294    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555296    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555299    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555301    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555304    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555306    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555309    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 04:49:56.561060 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555311    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555314    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555316    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555318    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555321    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555323    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555326    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555328    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555331    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555334    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555337    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555339    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555342    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555344    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555347    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555351    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555354    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555357    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555360    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555363    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 04:49:56.561563 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555365    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555368    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555371    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555373    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555375    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555378    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555380    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555383    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555385    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555387    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555390    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555392    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555394    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555397    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555399    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555401    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555404    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555406    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555409    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555411    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 04:49:56.562142 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555414    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555416    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555419    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555421    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555423    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555426    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:56.555428    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.555433    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.555548    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.557659    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.559027    2582 server.go:1019] "Starting client certificate rotation"
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.559122    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 04:49:56.562693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.559873    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 04:49:56.563009 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.562993    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 04:49:56.566136 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.566121    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 04:49:56.576765 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.576745    2582 log.go:25] "Validated CRI v1 runtime API"
Jul 17 04:49:56.579402 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.579385    2582 log.go:25] "Validated CRI v1 image API"
Jul 17 04:49:56.580137 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.580122    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 04:49:56.583601 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.583580    2582 fs.go:135] Filesystem UUIDs: map[718e5b72-b410-46fa-8d3e-82029cb589e0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 903524bb-107d-4a39-ba28-1fe64cb97c69:/dev/nvme0n1p3]
Jul 17 04:49:56.583654 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.583602    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 04:49:56.586840 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.586823    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 04:49:56.589611 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.589490    2582 manager.go:216] Machine: {Timestamp:2026-07-17 04:49:56.587602101 +0000 UTC m=+0.324664439 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101189 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec202c4ad43892c4f7b2807dc6c42d57 SystemUUID:ec202c4a-d438-92c4-f7b2-807dc6c42d57 BootID:5d801ca3-9185-472b-93d0-d78ea484992b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:b8:03:88:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:b8:03:88:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:6b:81:e4:cf:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 04:49:56.589611 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.589607    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 04:49:56.589714 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.589695    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 04:49:56.592190 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.592163    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 04:49:56.592338 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.592193    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 04:49:56.593451 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.593441    2582 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 04:49:56.593485 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.593454    2582 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 04:49:56.593485 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.593462    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 04:49:56.594045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.594034    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 04:49:56.595244 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.595233    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 04:49:56.599022 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599009    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 04:49:56.599138 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599130    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 04:49:56.599176 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599160    2582 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 04:49:56.599176 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599170    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 04:49:56.599229 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599185    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 04:49:56.599229 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599194    2582 kubelet.go:398] "Adding apiserver pod source"
Jul 17 04:49:56.599229 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599204    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 04:49:56.599846 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599816    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 04:49:56.599846 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.599850    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 04:49:56.604529 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.604506    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 04:49:56.606439 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.606422    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 04:49:56.606509 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.606454    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 04:49:56.608277 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608249    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608283    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608289    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608295    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608310    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608316    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608323    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 04:49:56.608326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608329    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 04:49:56.608511 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608336    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 04:49:56.608511 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608342    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 04:49:56.608511 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608350    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 04:49:56.608511 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608355    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 04:49:56.609002 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.608990    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 04:49:56.609002 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.609000    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 04:49:56.609412 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.609376    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 04:49:56.609412 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.609376    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 04:49:56.612241 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.612226    2582 server.go:1268] "Started kubelet"
Jul 17 04:49:56.612391 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.612341    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 04:49:56.612469 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.612424    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 04:49:56.613127 ip-10-0-140-139 systemd[1]: Started Kubernetes Kubelet.
Jul 17 04:49:56.613314 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.613116    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 04:49:56.614249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.614225    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8bqp"
Jul 17 04:49:56.614364 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.614311    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 04:49:56.614532 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.614519    2582 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 04:49:56.617586 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.617566    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 04:49:56.618765 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.618749    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 04:49:56.619461 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.619445    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8bqp"
Jul 17 04:49:56.619537 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.618647    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-139.ec2.internal.18c2fa7836a8e41a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-139.ec2.internal,UID:ip-10-0-140-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-139.ec2.internal,},FirstTimestamp:2026-07-17 04:49:56.612195354 +0000 UTC m=+0.349257681,LastTimestamp:2026-07-17 04:49:56.612195354 +0000 UTC m=+0.349257681,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-139.ec2.internal,}"
Jul 17 04:49:56.620765 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.620750    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 04:49:56.620765 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.620759    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 04:49:56.620877 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.620820    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:56.621369 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621344    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 04:49:56.621369 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621365    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 04:49:56.621504 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621343    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 04:49:56.621590 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621571    2582 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 04:49:56.621590 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621586    2582 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 04:49:56.621737 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621630    2582 factory.go:55] Registering systemd factory
Jul 17 04:49:56.621737 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621646    2582 factory.go:223] Registration of the systemd container factory successfully
Jul 17 04:49:56.621865 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621847    2582 factory.go:153] Registering CRI-O factory
Jul 17 04:49:56.621865 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621864    2582 factory.go:223] Registration of the crio container factory successfully
Jul 17 04:49:56.622034 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621918    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 04:49:56.622034 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621942    2582 factory.go:103] Registering Raw factory
Jul 17 04:49:56.622034 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.621969    2582 manager.go:1196] Started watching for new ooms in manager
Jul 17 04:49:56.622342 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.622325    2582 manager.go:349] Starting recovery of all containers
Jul 17 04:49:56.624769 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.624739    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 04:49:56.624875 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.624851    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 04:49:56.634007 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.633990    2582 manager.go:354] Recovery completed
Jul 17 04:49:56.635348 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.635318    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 17 04:49:56.638652 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.638634    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.640478 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.640461    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.640536 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.640494    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.640536 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.640505    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.641025 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641010    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 04:49:56.641025 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641024    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 04:49:56.641118 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641041    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 04:49:56.641118 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641054    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.641795 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641771    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.641867 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641800    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.641867 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.641811    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.642992 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.642980    2582 policy_none.go:49] "None policy: Start"
Jul 17 04:49:56.643033 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.642996    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 04:49:56.643033 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.643009    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 04:49:56.644936 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.644924    2582 policy_none.go:47] "Start"
Jul 17 04:49:56.683060 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.682853    2582 manager.go:337] "Starting Device Plugin manager"
Jul 17 04:49:56.683191 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.683102    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 04:49:56.683191 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.683116    2582 server.go:87] "Starting device plugin registration server"
Jul 17 04:49:56.683414 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.683401    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 04:49:56.683484 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.683417    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 04:49:56.683534 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.683519    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 04:49:56.683614 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.683598    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 04:49:56.683614 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.683611    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 04:49:56.684194 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.684173    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 04:49:56.684297 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.684225    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:56.722668 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.722633    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 04:49:56.729904 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.723977    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 04:49:56.729904 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.724016    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 04:49:56.729904 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.724043    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 04:49:56.729904 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.724088    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 04:49:56.729904 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.726024    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 04:49:56.784109 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.784075    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.785856 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.785836    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.785984 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.785877    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.785984 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.785890    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.785984 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.785922    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.793824 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.793760    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.793824 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.793787    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-139.ec2.internal\": node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:56.814067 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.814043    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:56.824834 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.824800    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"]
Jul 17 04:49:56.824925 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.824886    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.825834 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.825819    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.825930 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.825853    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.825930 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.825867    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.828069 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828052    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.828179 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.828226 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828197    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.828791 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828773    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.828897 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828774    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.828897 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828822    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.828897 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828833    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.828897 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828806    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.828897 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.828886    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.831081 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.831065    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.831130 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.831102    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 04:49:56.831839 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.831823    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 04:49:56.831945 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.831852    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 04:49:56.831945 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.831867    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeHasSufficientPID"
Jul 17 04:49:56.847847 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.847828    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-139.ec2.internal\" not found" node="ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.852556 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.852540    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-139.ec2.internal\" not found" node="ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.914310 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:56.914280    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:56.923378 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.923342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/804f47a9a03ca597698b8d2618e52e54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal\" (UID: \"804f47a9a03ca597698b8d2618e52e54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.923378 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.923382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/804f47a9a03ca597698b8d2618e52e54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal\" (UID: \"804f47a9a03ca597698b8d2618e52e54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:56.923507 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:56.923401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71c35c8b98f8ea365483d4a890e41d1a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-139.ec2.internal\" (UID: \"71c35c8b98f8ea365483d4a890e41d1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.015216 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.015187    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.023527 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.023503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/804f47a9a03ca597698b8d2618e52e54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal\" (UID: \"804f47a9a03ca597698b8d2618e52e54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.023590 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.023531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/804f47a9a03ca597698b8d2618e52e54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal\" (UID: \"804f47a9a03ca597698b8d2618e52e54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.023590 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.023550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71c35c8b98f8ea365483d4a890e41d1a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-139.ec2.internal\" (UID: \"71c35c8b98f8ea365483d4a890e41d1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.023590 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.023576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71c35c8b98f8ea365483d4a890e41d1a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-139.ec2.internal\" (UID: \"71c35c8b98f8ea365483d4a890e41d1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.023711 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.023623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/804f47a9a03ca597698b8d2618e52e54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal\" (UID: \"804f47a9a03ca597698b8d2618e52e54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.023711 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.023623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/804f47a9a03ca597698b8d2618e52e54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal\" (UID: \"804f47a9a03ca597698b8d2618e52e54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.116042 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.115964    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.151454 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.151421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.155154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.155140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.216653 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.216616    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.317144 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.317108    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.417691 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.417653    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.518119 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.518074    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.558483 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.558446    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 04:49:57.558966 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.558650    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 04:49:57.618828 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.618793    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.625528 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.625502    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 04:44:56 +0000 UTC" deadline="2028-04-25 04:52:31.787515094 +0000 UTC"
Jul 17 04:49:57.625528 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.625525    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15552h2m34.161992951s"
Jul 17 04:49:57.719845 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:57.719811    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-139.ec2.internal\" not found"
Jul 17 04:49:57.743772 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.743742    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 04:49:57.744729 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.744715    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 04:49:57.759036 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:57.758995    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804f47a9a03ca597698b8d2618e52e54.slice/crio-0ebd9bf45d929f12b97b324a8d4ba9f142dd2eede5cf48c33e800554d64be993 WatchSource:0}: Error finding container 0ebd9bf45d929f12b97b324a8d4ba9f142dd2eede5cf48c33e800554d64be993: Status 404 returned error can't find the container with id 0ebd9bf45d929f12b97b324a8d4ba9f142dd2eede5cf48c33e800554d64be993
Jul 17 04:49:57.759376 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:57.759360    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c35c8b98f8ea365483d4a890e41d1a.slice/crio-9b87aae7342929053717e8f135983ef1589063998f88986a9bcb50e2c7291d9e WatchSource:0}: Error finding container 9b87aae7342929053717e8f135983ef1589063998f88986a9bcb50e2c7291d9e: Status 404 returned error can't find the container with id 9b87aae7342929053717e8f135983ef1589063998f88986a9bcb50e2c7291d9e
Jul 17 04:49:57.764360 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.764346    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 04:49:57.820903 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.820877    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.832446 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.832420    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 04:49:57.833227 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.833209    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal"
Jul 17 04:49:57.932965 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:57.932889    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 04:49:58.194221 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.194136    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 04:49:58.600553 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.600482    2582 apiserver.go:52] "Watching apiserver"
Jul 17 04:49:58.610092 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.610068    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 04:49:58.610534 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.610516    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-ncz2w","openshift-dns/node-resolver-6s8kt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-d282m","openshift-network-diagnostics/network-check-target-wlb4b","openshift-ovn-kubernetes/ovnkube-node-nxz9q","kube-system/global-pull-secret-syncer-drpqq","kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp","openshift-image-registry/node-ca-hmd9s","openshift-multus/multus-qf4nq","openshift-multus/network-metrics-daemon-9nwkj","openshift-network-operator/iptables-alerter-nwnpq","kube-system/konnectivity-agent-d9x2p"]
Jul 17 04:49:58.613329 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.613310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.613435 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.613387    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:49:58.615306 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.615232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.618690 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.618666    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 04:49:58.618792 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.618732    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.618911 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.618895    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.618965 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.618923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.619161 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.619148    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-znb26\""
Jul 17 04:49:58.619662 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.619637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.621352 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.621334    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 04:49:58.621459 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.621439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kkhjl\""
Jul 17 04:49:58.621522 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.621515    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 04:49:58.622282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.621760    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 04:49:58.622282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:49:58.622282 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.622055    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:49:58.622483 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622349    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.622483 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622358    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 04:49:58.622483 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622375    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.622483 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622405    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58jsd\""
Jul 17 04:49:58.622648 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622629    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 04:49:58.622760 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.622639    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 04:49:58.625080 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.624821    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.628242 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.627559    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.628242 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.627774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 04:49:58.628242 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.627835    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.628697 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.628348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.630092 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5be18d5-6b55-42fa-8669-03ac30ab7f33-dbus\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.630196 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-cnibin\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630196 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz82\" (UniqueName: \"kubernetes.io/projected/2c6552ab-867d-4bc6-8958-d12371149d7d-kube-api-access-5zz82\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630196 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4523850-33cb-454a-a754-0d35581863a8-iptables-alerter-script\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.630196 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94cabee3-1dc5-4b2f-97be-f5d095b26776-konnectivity-ca\") pod \"konnectivity-agent-d9x2p\" (UID: \"94cabee3-1dc5-4b2f-97be-f5d095b26776\") " pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4523850-33cb-454a-a754-0d35581863a8-host-slash\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpwm\" (UniqueName: \"kubernetes.io/projected/b4523850-33cb-454a-a754-0d35581863a8-kube-api-access-jqpwm\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94cabee3-1dc5-4b2f-97be-f5d095b26776-agent-certs\") pod \"konnectivity-agent-d9x2p\" (UID: \"94cabee3-1dc5-4b2f-97be-f5d095b26776\") " pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.630413 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630372    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 04:49:58.630725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 04:49:58.630725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-os-release\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5be18d5-6b55-42fa-8669-03ac30ab7f33-kubelet-config\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.630725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630666    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.630725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-system-cni-dir\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630999 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630999 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.630999 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630766    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4hb6h\""
Jul 17 04:49:58.630999 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.630695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 04:49:58.631181 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.631004    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 04:49:58.631181 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.631054    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.631302 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.631192    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lxwc4\""
Jul 17 04:49:58.633123 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.632855    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 04:49:58.633740 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.633717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.635481 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.635461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.638808 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.638787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.638999 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.638981    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.639175 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.639159    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.639275 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.639230    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.639400 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.639383    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xrqkr\""
Jul 17 04:49:58.639485 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.639468    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wpst4\""
Jul 17 04:49:58.640175 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.640156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:58.640272 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.640235    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:49:58.640832 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.640795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.643061 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.643041    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dfx6f\""
Jul 17 04:49:58.643387 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.643334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.644494 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.644468    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 04:49:58.647175 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.647153    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 04:49:58.647431 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.647413    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 04:49:58.647728 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.647695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 04:49:58.648001 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.647985    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8kszg\""
Jul 17 04:49:58.660558 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.660535    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dpt2q"
Jul 17 04:49:58.691254 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.691226    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dpt2q"
Jul 17 04:49:58.722342 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.722310    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 04:49:58.724525 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.724504    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 04:49:58.729317 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.729238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal" event={"ID":"804f47a9a03ca597698b8d2618e52e54","Type":"ContainerStarted","Data":"0ebd9bf45d929f12b97b324a8d4ba9f142dd2eede5cf48c33e800554d64be993"}
Jul 17 04:49:58.730507 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.730478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal" event={"ID":"71c35c8b98f8ea365483d4a890e41d1a","Type":"ContainerStarted","Data":"9b87aae7342929053717e8f135983ef1589063998f88986a9bcb50e2c7291d9e"}
Jul 17 04:49:58.731697 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-registration-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.731806 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.731806 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-sys-fs\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.731806 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxfl\" (UniqueName: \"kubernetes.io/projected/3c56e369-037f-4c68-b326-7aed0d693cba-kube-api-access-8cxfl\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.731806 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-multus-certs\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.731806 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-log-socket\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-run-ovn-kubernetes\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8psd\" (UniqueName: \"kubernetes.io/projected/90ebd0bd-0591-47a7-b42f-79c949d53fe7-kube-api-access-x8psd\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-kubelet\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-conf-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-var-lib-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.731978    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.731991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-env-overrides\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.732045    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:49:59.232023689 +0000 UTC m=+2.969086024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-var-lib-kubelet\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce414902-197a-417e-8605-a88717e915ec-multus-daemon-config\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-slash\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-etc-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-cni-bin\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzmv\" (UniqueName: \"kubernetes.io/projected/73058375-9228-4e88-93c4-3875b9c1ba27-kube-api-access-8nzmv\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysctl-d\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qz5k\" (UniqueName: \"kubernetes.io/projected/cc964d46-6df9-4848-8b80-ce5423eab530-kube-api-access-8qz5k\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4523850-33cb-454a-a754-0d35581863a8-iptables-alerter-script\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94cabee3-1dc5-4b2f-97be-f5d095b26776-konnectivity-ca\") pod \"konnectivity-agent-d9x2p\" (UID: \"94cabee3-1dc5-4b2f-97be-f5d095b26776\") " pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-system-cni-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-cni-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732463 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-ovnkube-config\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-device-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-os-release\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-socket-dir-parent\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-ovn\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysctl-conf\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-sys\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-host\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-kubernetes\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-run\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-tuned\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1696bf1f-c106-40a0-b368-a5e390d7431b-tmp\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p99m\" (UniqueName: \"kubernetes.io/projected/1696bf1f-c106-40a0-b368-a5e390d7431b-kube-api-access-5p99m\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3efb5631-c960-43ec-86b8-3038e6486013-serviceca\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732905    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.732961 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-ovnkube-script-lib\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.733670 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.732938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-modprobe-d\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.733670 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-systemd\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.733670 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94cabee3-1dc5-4b2f-97be-f5d095b26776-konnectivity-ca\") pod \"konnectivity-agent-d9x2p\" (UID: \"94cabee3-1dc5-4b2f-97be-f5d095b26776\") " pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.733858 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.733982 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-cni-netd\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.733982 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4523850-33cb-454a-a754-0d35581863a8-iptables-alerter-script\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.733982 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.734108 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.733984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94cabee3-1dc5-4b2f-97be-f5d095b26776-agent-certs\") pod \"konnectivity-agent-d9x2p\" (UID: \"94cabee3-1dc5-4b2f-97be-f5d095b26776\") " pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.734108 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-cnibin\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.734108 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce414902-197a-417e-8605-a88717e915ec-cni-binary-copy\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.734108 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-hostroot\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.734253 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-kubelet\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.734253 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73058375-9228-4e88-93c4-3875b9c1ba27-ovn-node-metrics-cert\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.734253 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysconfig\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.734253 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5be18d5-6b55-42fa-8669-03ac30ab7f33-kubelet-config\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.734253 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-socket-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-etc-kubernetes\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734304    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c6552ab-867d-4bc6-8958-d12371149d7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734351    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5be18d5-6b55-42fa-8669-03ac30ab7f33-kubelet-config\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-cnibin\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-netns\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-systemd-units\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.734490 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-node-log\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-k8s-cni-cncf-io\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-cnibin\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-run-netns\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-lib-modules\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/3efb5631-c960-43ec-86b8-3038e6486013-kube-api-access-s6pck\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90ebd0bd-0591-47a7-b42f-79c949d53fe7-hosts-file\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4523850-33cb-454a-a754-0d35581863a8-host-slash\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpwm\" (UniqueName: \"kubernetes.io/projected/b4523850-33cb-454a-a754-0d35581863a8-kube-api-access-jqpwm\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-os-release\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4523850-33cb-454a-a754-0d35581863a8-host-slash\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.734873 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ebd0bd-0591-47a7-b42f-79c949d53fe7-tmp-dir\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rxh\" (UniqueName: \"kubernetes.io/projected/ce414902-197a-417e-8605-a88717e915ec-kube-api-access-48rxh\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-os-release\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-system-cni-dir\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-cni-multus\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.734978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6552ab-867d-4bc6-8958-d12371149d7d-system-cni-dir\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.735015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5be18d5-6b55-42fa-8669-03ac30ab7f33-dbus\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.735042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz82\" (UniqueName: \"kubernetes.io/projected/2c6552ab-867d-4bc6-8958-d12371149d7d-kube-api-access-5zz82\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.735077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-cni-bin\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.735178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-systemd\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.735193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5be18d5-6b55-42fa-8669-03ac30ab7f33-dbus\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:58.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.735202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3efb5631-c960-43ec-86b8-3038e6486013-host\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.738443 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.738402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94cabee3-1dc5-4b2f-97be-f5d095b26776-agent-certs\") pod \"konnectivity-agent-d9x2p\" (UID: \"94cabee3-1dc5-4b2f-97be-f5d095b26776\") " pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.741584 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.741507    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:49:58.741584 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.741532    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:49:58.741584 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.741547    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:49:58.741844 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.741601    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:49:59.241584311 +0000 UTC m=+2.978646646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:49:58.744249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.744226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz82\" (UniqueName: \"kubernetes.io/projected/2c6552ab-867d-4bc6-8958-d12371149d7d-kube-api-access-5zz82\") pod \"multus-additional-cni-plugins-d282m\" (UID: \"2c6552ab-867d-4bc6-8958-d12371149d7d\") " pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.744377 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.744356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpwm\" (UniqueName: \"kubernetes.io/projected/b4523850-33cb-454a-a754-0d35581863a8-kube-api-access-jqpwm\") pod \"iptables-alerter-nwnpq\" (UID: \"b4523850-33cb-454a-a754-0d35581863a8\") " pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.835561 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-cni-netd\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.835561 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-cnibin\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce414902-197a-417e-8605-a88717e915ec-cni-binary-copy\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-cni-netd\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-hostroot\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-cnibin\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-hostroot\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-kubelet\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-kubelet\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73058375-9228-4e88-93c4-3875b9c1ba27-ovn-node-metrics-cert\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysconfig\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.835813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-socket-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-etc-kubernetes\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-netns\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-systemd-units\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-node-log\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-k8s-cni-cncf-io\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.835995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-run-netns\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-lib-modules\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/3efb5631-c960-43ec-86b8-3038e6486013-kube-api-access-s6pck\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90ebd0bd-0591-47a7-b42f-79c949d53fe7-hosts-file\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836154 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ebd0bd-0591-47a7-b42f-79c949d53fe7-tmp-dir\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rxh\" (UniqueName: \"kubernetes.io/projected/ce414902-197a-417e-8605-a88717e915ec-kube-api-access-48rxh\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-cni-multus\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-cni-bin\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-systemd\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce414902-197a-417e-8605-a88717e915ec-cni-binary-copy\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3efb5631-c960-43ec-86b8-3038e6486013-host\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-registration-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-run-netns\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysconfig\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-sys-fs\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxfl\" (UniqueName: \"kubernetes.io/projected/3c56e369-037f-4c68-b326-7aed0d693cba-kube-api-access-8cxfl\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-multus-certs\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-log-socket\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-socket-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-run-ovn-kubernetes\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.836744 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-etc-kubernetes\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8psd\" (UniqueName: \"kubernetes.io/projected/90ebd0bd-0591-47a7-b42f-79c949d53fe7-kube-api-access-x8psd\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-kubelet\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-netns\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-conf-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-var-lib-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-systemd-units\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-env-overrides\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-sys-fs\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-node-log\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-k8s-cni-cncf-io\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-run-multus-certs\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ebd0bd-0591-47a7-b42f-79c949d53fe7-tmp-dir\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-cni-multus\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-cni-bin\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.836869    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-systemd\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.837539 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3efb5631-c960-43ec-86b8-3038e6486013-host\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:58.836965    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:49:59.336948273 +0000 UTC m=+3.074010605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-log-socket\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-registration-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-host-var-lib-kubelet\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-var-lib-kubelet\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce414902-197a-417e-8605-a88717e915ec-multus-daemon-config\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-var-lib-kubelet\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-slash\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-etc-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90ebd0bd-0591-47a7-b42f-79c949d53fe7-hosts-file\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-cni-bin\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzmv\" (UniqueName: \"kubernetes.io/projected/73058375-9228-4e88-93c4-3875b9c1ba27-kube-api-access-8nzmv\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-etc-selinux\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysctl-d\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.838379 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-var-lib-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qz5k\" (UniqueName: \"kubernetes.io/projected/cc964d46-6df9-4848-8b80-ce5423eab530-kube-api-access-8qz5k\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-system-cni-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-conf-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysctl-d\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.836964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-lib-modules\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-system-cni-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-run-ovn-kubernetes\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-cni-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-ovnkube-config\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-slash\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-device-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-etc-openvswitch\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-os-release\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-socket-dir-parent\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-os-release\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.839150 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-ovn\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysctl-conf\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-cni-dir\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-sys\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-host\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-kubernetes\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-run\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-tuned\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1696bf1f-c106-40a0-b368-a5e390d7431b-tmp\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p99m\" (UniqueName: \"kubernetes.io/projected/1696bf1f-c106-40a0-b368-a5e390d7431b-kube-api-access-5p99m\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3efb5631-c960-43ec-86b8-3038e6486013-serviceca\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-ovnkube-script-lib\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-modprobe-d\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-env-overrides\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-host\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-host-cni-bin\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-systemd\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-kubernetes\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.837975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-run\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-ovnkube-config\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c56e369-037f-4c68-b326-7aed0d693cba-device-dir\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce414902-197a-417e-8605-a88717e915ec-multus-socket-dir-parent\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-sys\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-systemd\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-sysctl-conf\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73058375-9228-4e88-93c4-3875b9c1ba27-run-ovn\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73058375-9228-4e88-93c4-3875b9c1ba27-ovnkube-script-lib\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3efb5631-c960-43ec-86b8-3038e6486013-serviceca\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.838858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73058375-9228-4e88-93c4-3875b9c1ba27-ovn-node-metrics-cert\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.839118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-modprobe-d\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.840146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1696bf1f-c106-40a0-b368-a5e390d7431b-etc-tuned\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.840339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce414902-197a-417e-8605-a88717e915ec-multus-daemon-config\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.840931 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.840794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1696bf1f-c106-40a0-b368-a5e390d7431b-tmp\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.848021 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.847951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p99m\" (UniqueName: \"kubernetes.io/projected/1696bf1f-c106-40a0-b368-a5e390d7431b-kube-api-access-5p99m\") pod \"tuned-ncz2w\" (UID: \"1696bf1f-c106-40a0-b368-a5e390d7431b\") " pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.848340 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.848312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qz5k\" (UniqueName: \"kubernetes.io/projected/cc964d46-6df9-4848-8b80-ce5423eab530-kube-api-access-8qz5k\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:58.848803 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.848752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/3efb5631-c960-43ec-86b8-3038e6486013-kube-api-access-s6pck\") pod \"node-ca-hmd9s\" (UID: \"3efb5631-c960-43ec-86b8-3038e6486013\") " pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:58.849211 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.849188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8psd\" (UniqueName: \"kubernetes.io/projected/90ebd0bd-0591-47a7-b42f-79c949d53fe7-kube-api-access-x8psd\") pod \"node-resolver-6s8kt\" (UID: \"90ebd0bd-0591-47a7-b42f-79c949d53fe7\") " pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.850350 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.850324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxfl\" (UniqueName: \"kubernetes.io/projected/3c56e369-037f-4c68-b326-7aed0d693cba-kube-api-access-8cxfl\") pod \"aws-ebs-csi-driver-node-g4fjp\" (UID: \"3c56e369-037f-4c68-b326-7aed0d693cba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.851147 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.851091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzmv\" (UniqueName: \"kubernetes.io/projected/73058375-9228-4e88-93c4-3875b9c1ba27-kube-api-access-8nzmv\") pod \"ovnkube-node-nxz9q\" (UID: \"73058375-9228-4e88-93c4-3875b9c1ba27\") " pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.851527 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.851490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rxh\" (UniqueName: \"kubernetes.io/projected/ce414902-197a-417e-8605-a88717e915ec-kube-api-access-48rxh\") pod \"multus-qf4nq\" (UID: \"ce414902-197a-417e-8605-a88717e915ec\") " pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:58.935528 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.935479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nwnpq"
Jul 17 04:49:58.941467 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.941431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:49:58.954140 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.954105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d282m"
Jul 17 04:49:58.958145 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.958121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:49:58.964929 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.964910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp"
Jul 17 04:49:58.978583 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.978553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6s8kt"
Jul 17 04:49:58.984303 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.984279    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ncz2w"
Jul 17 04:49:58.993037 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:58.993017    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qf4nq"
Jul 17 04:49:59.000794 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.000769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmd9s"
Jul 17 04:49:59.240105 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.240013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:49:59.240287 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.240164    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:49:59.240287 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.240231    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:00.240214669 +0000 UTC m=+3.977277071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:49:59.341229 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.341194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:59.341229 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.341234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:49:59.341431 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.341399    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:49:59.341502 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.341488    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:00.341465188 +0000 UTC m=+4.078527516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:49:59.341557 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.341408    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:49:59.341557 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.341525    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:49:59.341557 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.341542    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:49:59.341660 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.341591    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:50:00.341576646 +0000 UTC m=+4.078638960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:49:59.685115 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:59.684890    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6552ab_867d_4bc6_8958_d12371149d7d.slice/crio-f89721722fa746e0341c41b2e9859d49164b758888ec9c93e17cb56ce7063a7d WatchSource:0}: Error finding container f89721722fa746e0341c41b2e9859d49164b758888ec9c93e17cb56ce7063a7d: Status 404 returned error can't find the container with id f89721722fa746e0341c41b2e9859d49164b758888ec9c93e17cb56ce7063a7d
Jul 17 04:49:59.689427 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:59.689405    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1696bf1f_c106_40a0_b368_a5e390d7431b.slice/crio-d0e1dab044e600508e6eb1e763cfa6d026857a1a73a1cea4b4278021f919a330 WatchSource:0}: Error finding container d0e1dab044e600508e6eb1e763cfa6d026857a1a73a1cea4b4278021f919a330: Status 404 returned error can't find the container with id d0e1dab044e600508e6eb1e763cfa6d026857a1a73a1cea4b4278021f919a330
Jul 17 04:49:59.690861 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:59.690541    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73058375_9228_4e88_93c4_3875b9c1ba27.slice/crio-0c7c85406177747745ba81eb7bcaf714bb3e2471802b23f88c99246135e6a942 WatchSource:0}: Error finding container 0c7c85406177747745ba81eb7bcaf714bb3e2471802b23f88c99246135e6a942: Status 404 returned error can't find the container with id 0c7c85406177747745ba81eb7bcaf714bb3e2471802b23f88c99246135e6a942
Jul 17 04:49:59.691303 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:59.691063    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efb5631_c960_43ec_86b8_3038e6486013.slice/crio-4bb1cbb3ff587e64ba398eaf59592948f9c4c35348ae0be64ee607149c16a6ff WatchSource:0}: Error finding container 4bb1cbb3ff587e64ba398eaf59592948f9c4c35348ae0be64ee607149c16a6ff: Status 404 returned error can't find the container with id 4bb1cbb3ff587e64ba398eaf59592948f9c4c35348ae0be64ee607149c16a6ff
Jul 17 04:49:59.694923 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.694720    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 04:44:58 +0000 UTC" deadline="2028-02-12 00:08:46.919862153 +0000 UTC"
Jul 17 04:49:59.694923 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.694761    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13795h18m47.225104821s"
Jul 17 04:49:59.714497 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:59.714469    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ebd0bd_0591_47a7_b42f_79c949d53fe7.slice/crio-2c36e2ba66d5addec560fd2988672b37ad205e41902c7fc18c5a7e50fea27e25 WatchSource:0}: Error finding container 2c36e2ba66d5addec560fd2988672b37ad205e41902c7fc18c5a7e50fea27e25: Status 404 returned error can't find the container with id 2c36e2ba66d5addec560fd2988672b37ad205e41902c7fc18c5a7e50fea27e25
Jul 17 04:49:59.715845 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:49:59.715812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce414902_197a_417e_8605_a88717e915ec.slice/crio-dcc1c14d0ceb3bd5e3f9a8859eb4bd0d7befc05d87d7bcc0e11e9da66b4f6d37 WatchSource:0}: Error finding container dcc1c14d0ceb3bd5e3f9a8859eb4bd0d7befc05d87d7bcc0e11e9da66b4f6d37: Status 404 returned error can't find the container with id dcc1c14d0ceb3bd5e3f9a8859eb4bd0d7befc05d87d7bcc0e11e9da66b4f6d37
Jul 17 04:49:59.724232 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.724213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:49:59.724376 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:49:59.724358    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:49:59.733222 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.733192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qf4nq" event={"ID":"ce414902-197a-417e-8605-a88717e915ec","Type":"ContainerStarted","Data":"dcc1c14d0ceb3bd5e3f9a8859eb4bd0d7befc05d87d7bcc0e11e9da66b4f6d37"}
Jul 17 04:49:59.734469 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.734444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nwnpq" event={"ID":"b4523850-33cb-454a-a754-0d35581863a8","Type":"ContainerStarted","Data":"5c8decb6d870ac699a0e90b8b1f058b840c3adbfb88de256f3cd9b6af98e081b"}
Jul 17 04:49:59.735762 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.735738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6s8kt" event={"ID":"90ebd0bd-0591-47a7-b42f-79c949d53fe7","Type":"ContainerStarted","Data":"2c36e2ba66d5addec560fd2988672b37ad205e41902c7fc18c5a7e50fea27e25"}
Jul 17 04:49:59.736851 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.736827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9x2p" event={"ID":"94cabee3-1dc5-4b2f-97be-f5d095b26776","Type":"ContainerStarted","Data":"386b838e9848f011ede1e401b75db70e0e33d80675c020dff14feb368b545f49"}
Jul 17 04:49:59.738044 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.738014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp" event={"ID":"3c56e369-037f-4c68-b326-7aed0d693cba","Type":"ContainerStarted","Data":"4166778fcf03190ab1fcaa981f6dce7ad2c415b4594414b24eda38d75ae3ce57"}
Jul 17 04:49:59.739473 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.739445    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmd9s" event={"ID":"3efb5631-c960-43ec-86b8-3038e6486013","Type":"ContainerStarted","Data":"4bb1cbb3ff587e64ba398eaf59592948f9c4c35348ae0be64ee607149c16a6ff"}
Jul 17 04:49:59.742499 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.742474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"0c7c85406177747745ba81eb7bcaf714bb3e2471802b23f88c99246135e6a942"}
Jul 17 04:49:59.746114 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.746064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ncz2w" event={"ID":"1696bf1f-c106-40a0-b368-a5e390d7431b","Type":"ContainerStarted","Data":"d0e1dab044e600508e6eb1e763cfa6d026857a1a73a1cea4b4278021f919a330"}
Jul 17 04:49:59.749888 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:49:59.749863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerStarted","Data":"f89721722fa746e0341c41b2e9859d49164b758888ec9c93e17cb56ce7063a7d"}
Jul 17 04:50:00.249233 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.249193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:00.249423 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.249391    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:00.249496 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.249460    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:02.249440837 +0000 UTC m=+5.986503155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.349597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.349652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.349830    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.349848    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.349872    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.349935    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:50:02.349913931 +0000 UTC m=+6.086976259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.350345    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:00.350432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.350396    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:02.350381384 +0000 UTC m=+6.087443701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:00.695958 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.695854    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 04:44:58 +0000 UTC" deadline="2028-02-04 13:52:31.913210639 +0000 UTC"
Jul 17 04:50:00.695958 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.695889    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13617h2m31.217326221s"
Jul 17 04:50:00.725436 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.725357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:00.725602 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.725503    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:00.725957 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.725905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:00.726042 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:00.726001    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:00.780152 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.780108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal" event={"ID":"71c35c8b98f8ea365483d4a890e41d1a","Type":"ContainerStarted","Data":"ca84fc73bd173ee9923e827e68f1dee2c1048cadec8a0a4f0d1c854244a021fa"}
Jul 17 04:50:00.793522 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.793485    2582 generic.go:356] "Generic (PLEG): container finished" podID="804f47a9a03ca597698b8d2618e52e54" containerID="6e845f616b9626a047d0420f9b6484fd7157013f920273b7241180b5dba1c019" exitCode=0
Jul 17 04:50:00.793684 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.793544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal" event={"ID":"804f47a9a03ca597698b8d2618e52e54","Type":"ContainerDied","Data":"6e845f616b9626a047d0420f9b6484fd7157013f920273b7241180b5dba1c019"}
Jul 17 04:50:00.799309 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:00.799239    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-139.ec2.internal" podStartSLOduration=3.799219887 podStartE2EDuration="3.799219887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:00.798288704 +0000 UTC m=+4.535351040" watchObservedRunningTime="2026-07-17 04:50:00.799219887 +0000 UTC m=+4.536282224"
Jul 17 04:50:01.724577 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:01.724531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:01.725059 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:01.724695    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:01.805904 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:01.805792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal" event={"ID":"804f47a9a03ca597698b8d2618e52e54","Type":"ContainerStarted","Data":"08dff2b8019a3ca6f690955faab330de8c4c769d5f8eab4c71769ec07e7f3480"}
Jul 17 04:50:02.279821 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:02.279206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:02.279821 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.279347    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:02.279821 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.279432    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:06.27941134 +0000 UTC m=+10.016473658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:02.379798 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:02.379762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:02.379983 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:02.379817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:02.380109 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.380002    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:50:02.380109 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.380020    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:50:02.380109 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.380034    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:02.380109 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.380090    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:50:06.380072583 +0000 UTC m=+10.117134900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:02.380354 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.380164    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:02.380354 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.380199    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:06.380188128 +0000 UTC m=+10.117250444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:02.727176 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:02.726388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:02.727176 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.726521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:02.727176 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:02.726570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:02.727176 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:02.726697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:03.724784 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:03.724280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:03.724784 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:03.724430    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:04.725043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:04.725016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:04.725043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:04.725034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:04.725555 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:04.725123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:04.725555 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:04.725182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:05.724820 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:05.724784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:05.724987 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:05.724940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:06.314403 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:06.313817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:06.314403 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.313976    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:06.314403 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.314043    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:14.31402377 +0000 UTC m=+18.051086085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:06.414684 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:06.414648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:06.414860 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:06.414698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:06.414860 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.414828    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:50:06.414860 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.414844    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:50:06.414860 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.414853    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:06.415013 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.414903    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:50:14.414885966 +0000 UTC m=+18.151948303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:06.415013 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.414954    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:06.415013 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.414977    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:14.414970221 +0000 UTC m=+18.152032534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:06.725288 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:06.725188    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:06.725450 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.725313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:06.725774 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:06.725754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:06.725871 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:06.725854    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:07.725177 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:07.724971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:07.725177 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:07.725123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:08.728882 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:08.728209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:08.728882 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:08.728353    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:08.728882 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:08.728713    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:08.728882 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:08.728841    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:09.725143 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:09.725104    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:09.725341 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:09.725251    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:10.725100 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:10.725013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:10.725100 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:10.725037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:10.725614 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:10.725143    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:10.725614 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:10.725288    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:11.725197 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:11.725158    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:11.725625 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:11.725320    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:12.724513 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:12.724471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:12.724708 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:12.724479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:12.724708 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:12.724622    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:12.724708 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:12.724681    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:13.724580 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:13.724526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:13.725109 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:13.724680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:14.379644 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:14.379545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:14.379816 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.379710    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:14.379816 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.379788    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:30.379767837 +0000 UTC m=+34.116830156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:14.480736 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:14.480699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:14.480736 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:14.480743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:14.480964 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.480869    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:14.480964 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.480890    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:50:14.480964 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.480906    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:50:14.480964 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.480918    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:14.480964 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.480958    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:30.480935213 +0000 UTC m=+34.217997555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:14.481169 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.480978    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:50:30.480969103 +0000 UTC m=+34.218031417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:14.725488 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:14.724976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:14.725488 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:14.724993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:14.725488 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.725119    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:14.725488 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:14.725179    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:15.724664 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:15.724624    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:15.724842 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:15.724777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:16.727737 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:16.727704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:16.728225 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:16.727831    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:16.728225 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:16.727904    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:16.728225 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:16.727989    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:17.725207 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:17.725170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:17.725406 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:17.725319    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:18.724755 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:18.724732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:18.725066 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:18.724732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:18.725066 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:18.724896    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:18.725066 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:18.725009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:19.724915 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.724475    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:19.725705 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:19.725062    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:19.844780 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.844750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9x2p" event={"ID":"94cabee3-1dc5-4b2f-97be-f5d095b26776","Type":"ContainerStarted","Data":"ee4806c20cffe50e292d691da30baa6363c19a539cd9f4b1bc6cc14437eaaaad"}
Jul 17 04:50:19.846145 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.846124    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp" event={"ID":"3c56e369-037f-4c68-b326-7aed0d693cba","Type":"ContainerStarted","Data":"77ba6cd7ed8a73cc1bd79ba27f47a9c91a1d0da36cda8dc1b13b4082b42c9931"}
Jul 17 04:50:19.847625 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.847606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmd9s" event={"ID":"3efb5631-c960-43ec-86b8-3038e6486013","Type":"ContainerStarted","Data":"ebff4bdcd08c14ea7e4b502ee1750ac4569606d8b9868abd77e451431f76e044"}
Jul 17 04:50:19.850293 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850277    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:50:19.850606 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850589    2582 generic.go:356] "Generic (PLEG): container finished" podID="73058375-9228-4e88-93c4-3875b9c1ba27" containerID="d8bed5e1d573a265a3fc9b8d0413690bfee678cc71bcc9f51b83ed1a790ccecd" exitCode=1
Jul 17 04:50:19.850662 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"d775f1558feebed075a896d526d6c26f3149fd798abfc4c6527a247af1bcdeaf"}
Jul 17 04:50:19.850715 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"c3edc376f43ff57059429de1368ff7717f78156aa86a082b66ecaed571c28557"}
Jul 17 04:50:19.850766 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"13fae6aa2b989e504adc920a6415735d9fa9c06bbe33c12867447f810a8134b3"}
Jul 17 04:50:19.850820 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"e36bc64a4e54cc18344fc8e7414cf42c83612020b9a66c94b700a4a8993f42be"}
Jul 17 04:50:19.850820 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerDied","Data":"d8bed5e1d573a265a3fc9b8d0413690bfee678cc71bcc9f51b83ed1a790ccecd"}
Jul 17 04:50:19.850905 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.850832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"0d9c552c0b0bb1e365bbb679be62be5aeef1fc5dbca27e2522b6aaf769987f6a"}
Jul 17 04:50:19.851909 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.851887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ncz2w" event={"ID":"1696bf1f-c106-40a0-b368-a5e390d7431b","Type":"ContainerStarted","Data":"f1d9450bc99d5f28d56700424087a265899a1929a67a958b7c8232a6a1812468"}
Jul 17 04:50:19.853252 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.853233    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6552ab-867d-4bc6-8958-d12371149d7d" containerID="246fe01a8526c55e670d6fa74f816855eb83c4566f73c2301df228546ef1f86d" exitCode=0
Jul 17 04:50:19.853350 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.853282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerDied","Data":"246fe01a8526c55e670d6fa74f816855eb83c4566f73c2301df228546ef1f86d"}
Jul 17 04:50:19.854706 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.854689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qf4nq" event={"ID":"ce414902-197a-417e-8605-a88717e915ec","Type":"ContainerStarted","Data":"503c868231814dbd6fd8e12b38f441d564b541156f9e53064f03e48a1d0ec142"}
Jul 17 04:50:19.855726 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.855706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6s8kt" event={"ID":"90ebd0bd-0591-47a7-b42f-79c949d53fe7","Type":"ContainerStarted","Data":"5f779ad03d526813ca4d30969e0dfd6972da797820c2f1ac9081c6fa962a3a07"}
Jul 17 04:50:19.859616 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.859582    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-139.ec2.internal" podStartSLOduration=22.859572366 podStartE2EDuration="22.859572366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:01.821848247 +0000 UTC m=+5.558910682" watchObservedRunningTime="2026-07-17 04:50:19.859572366 +0000 UTC m=+23.596634701"
Jul 17 04:50:19.875634 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.875591    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d9x2p" podStartSLOduration=4.87687943 podStartE2EDuration="23.875570198s" totalImagesPullingTime="18.998690768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:19.860156112 +0000 UTC m=+23.597218448" watchObservedRunningTime="2026-07-17 04:50:19.875570198 +0000 UTC m=+23.612632536"
Jul 17 04:50:19.891364 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.891316    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ncz2w" podStartSLOduration=4.832188322 podStartE2EDuration="23.891299913s" totalImagesPullingTime="19.059111591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:19.875697013 +0000 UTC m=+23.612759350" watchObservedRunningTime="2026-07-17 04:50:19.891299913 +0000 UTC m=+23.628362245"
Jul 17 04:50:19.891804 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.891783    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qf4nq" podStartSLOduration=4.81564715 podStartE2EDuration="23.891775412s" totalImagesPullingTime="19.076128262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:19.891152581 +0000 UTC m=+23.628214917" watchObservedRunningTime="2026-07-17 04:50:19.891775412 +0000 UTC m=+23.628837745"
Jul 17 04:50:19.904732 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.904687    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hmd9s" podStartSLOduration=3.881930346 podStartE2EDuration="22.904668381s" totalImagesPullingTime="19.022738035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:19.904306755 +0000 UTC m=+23.641369091" watchObservedRunningTime="2026-07-17 04:50:19.904668381 +0000 UTC m=+23.641730717"
Jul 17 04:50:19.936609 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:19.936560    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6s8kt" podStartSLOduration=4.881527649 podStartE2EDuration="23.936536651s" totalImagesPullingTime="19.055009002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:19.936336107 +0000 UTC m=+23.673398443" watchObservedRunningTime="2026-07-17 04:50:19.936536651 +0000 UTC m=+23.673598986"
Jul 17 04:50:20.137514 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.137485    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 04:50:20.695982 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.695871    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T04:50:20.137504647Z","UUID":"e4924cd2-943f-4d9c-8cb0-cbbe79ce872f","Handler":null,"Name":"","Endpoint":""}
Jul 17 04:50:20.697724 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.697699    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 04:50:20.697840 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.697739    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 04:50:20.725952 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.725918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:20.726495 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:20.726061    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:20.726495 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.726208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:20.726495 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:20.726343    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:20.860355 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.860277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nwnpq" event={"ID":"b4523850-33cb-454a-a754-0d35581863a8","Type":"ContainerStarted","Data":"1c366752946d42e4f1ba7c6179d1430b4c7f03f1d70bfc81b56996f16d7f8314"}
Jul 17 04:50:20.862676 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.862647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp" event={"ID":"3c56e369-037f-4c68-b326-7aed0d693cba","Type":"ContainerStarted","Data":"27822ac1baab58d6031bc5a301f96d66c0d1af224dcc2128172d74ac4e7d0917"}
Jul 17 04:50:20.874923 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:20.874878    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nwnpq" podStartSLOduration=5.813259694 podStartE2EDuration="24.874864685s" totalImagesPullingTime="19.061604991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:20.874613831 +0000 UTC m=+24.611676168" watchObservedRunningTime="2026-07-17 04:50:20.874864685 +0000 UTC m=+24.611927020"
Jul 17 04:50:21.725056 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:21.725018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:21.725244 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:21.725148    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:21.867543 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:21.867435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp" event={"ID":"3c56e369-037f-4c68-b326-7aed0d693cba","Type":"ContainerStarted","Data":"016e9807e941f75c676c2c174c0f229041e66e39962c811eb36469a38061fcef"}
Jul 17 04:50:21.871403 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:21.871372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:50:21.871816 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:21.871788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"82964f83f5907625406e8ffb941e61b4257afa6d780d6a6d2fe2b745ca089ace"}
Jul 17 04:50:21.886795 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:21.886098    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g4fjp" podStartSLOduration=4.667817919 podStartE2EDuration="25.886076622s" totalImagesPullingTime="21.218258703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:21.885937767 +0000 UTC m=+25.623000104" watchObservedRunningTime="2026-07-17 04:50:21.886076622 +0000 UTC m=+25.623138957"
Jul 17 04:50:22.118378 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:22.118291    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:50:22.118893 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:22.118864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:50:22.725019 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:22.724992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:22.725199 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:22.724992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:22.725199 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:22.725124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:22.725199 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:22.725166    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:23.567278 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:23.567042    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:50:23.567725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:23.567707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9x2p"
Jul 17 04:50:23.725185 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:23.725138    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:23.725373 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:23.725274    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:24.725360 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.725277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:24.726129 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.725289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:24.726129 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:24.725414    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:24.726129 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:24.725462    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:24.881204 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.881031    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:50:24.881595 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.881565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"c7a59df3cc45363fcecc257544756c82252d555c4a9be939161d2fa80ea49c36"}
Jul 17 04:50:24.881913 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.881893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:50:24.882032 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.881966    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:50:24.882032 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.881997    2582 scope.go:117] "RemoveContainer" containerID="d8bed5e1d573a265a3fc9b8d0413690bfee678cc71bcc9f51b83ed1a790ccecd"
Jul 17 04:50:24.882032 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.882016    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:50:24.883693 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.883668    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6552ab-867d-4bc6-8958-d12371149d7d" containerID="a953176d7736398bbdc655078b19eb417828228076cbf64dda22d43d83ab6070" exitCode=0
Jul 17 04:50:24.883781 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.883711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerDied","Data":"a953176d7736398bbdc655078b19eb417828228076cbf64dda22d43d83ab6070"}
Jul 17 04:50:24.899759 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.899716    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:50:24.899980 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:24.899962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:50:25.725356 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:25.725304    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:25.725803 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:25.725459    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:25.889895 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:25.889810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:50:25.890290 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:25.890242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" event={"ID":"73058375-9228-4e88-93c4-3875b9c1ba27","Type":"ContainerStarted","Data":"c6134e9fa1f0aaee6241cc46f4b1629f746472d0be2069347ad507771471b06f"}
Jul 17 04:50:25.892246 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:25.892221    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6552ab-867d-4bc6-8958-d12371149d7d" containerID="e17545aff6bdad4c57d36a92877273e6934dcc9bce2ae3e1da173772767e8089" exitCode=0
Jul 17 04:50:25.892372 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:25.892291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerDied","Data":"e17545aff6bdad4c57d36a92877273e6934dcc9bce2ae3e1da173772767e8089"}
Jul 17 04:50:25.918697 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:25.918651    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q" podStartSLOduration=10.798527485 podStartE2EDuration="29.918636841s" totalImagesPullingTime="19.120109356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:25.918177094 +0000 UTC m=+29.655239450" watchObservedRunningTime="2026-07-17 04:50:25.918636841 +0000 UTC m=+29.655699176"
Jul 17 04:50:26.471671 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.471639    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drpqq"]
Jul 17 04:50:26.471836 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.471753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:26.471892 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:26.471840    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:26.474479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.474450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlb4b"]
Jul 17 04:50:26.474607 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.474565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:26.474686 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:26.474665    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:26.475054 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.475030    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9nwkj"]
Jul 17 04:50:26.475173 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.475110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:26.475228 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:26.475206    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:26.897054 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.897020    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6552ab-867d-4bc6-8958-d12371149d7d" containerID="fdf020feb605cb09cd9eba3bffea89c63916661d8661ebf8405115cdab906511" exitCode=0
Jul 17 04:50:26.897454 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:26.897113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerDied","Data":"fdf020feb605cb09cd9eba3bffea89c63916661d8661ebf8405115cdab906511"}
Jul 17 04:50:27.724473 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:27.724440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:27.724645 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:27.724585    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:28.728735 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:28.728504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:28.729141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:28.728512    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:28.729141 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:28.728843    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:28.729141 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:28.728918    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:29.724814 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:29.724778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:29.725070 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:29.724917    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drpqq" podUID="b5be18d5-6b55-42fa-8669-03ac30ab7f33"
Jul 17 04:50:30.399376 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:30.399331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:30.399934 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.399466    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:30.399934 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.399529    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret podName:b5be18d5-6b55-42fa-8669-03ac30ab7f33 nodeName:}" failed. No retries permitted until 2026-07-17 04:51:02.399511713 +0000 UTC m=+66.136574027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret") pod "global-pull-secret-syncer-drpqq" (UID: "b5be18d5-6b55-42fa-8669-03ac30ab7f33") : object "kube-system"/"original-pull-secret" not registered
Jul 17 04:50:30.500592 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:30.500549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:30.500769 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:30.500611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:30.500769 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.500722    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:30.500855 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.500782    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 04:50:30.500855 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.500792    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs podName:cc964d46-6df9-4848-8b80-ce5423eab530 nodeName:}" failed. No retries permitted until 2026-07-17 04:51:02.500776217 +0000 UTC m=+66.237838531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs") pod "network-metrics-daemon-9nwkj" (UID: "cc964d46-6df9-4848-8b80-ce5423eab530") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 04:50:30.500855 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.500801    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 04:50:30.500855 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.500811    2582 projected.go:196] Error preparing data for projected volume kube-api-access-j4wvk for pod openshift-network-diagnostics/network-check-target-wlb4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:30.501032 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.500863    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk podName:f3fb9c3b-3508-423f-82e7-8f1edb47358c nodeName:}" failed. No retries permitted until 2026-07-17 04:51:02.500850015 +0000 UTC m=+66.237912333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4wvk" (UniqueName: "kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk") pod "network-check-target-wlb4b" (UID: "f3fb9c3b-3508-423f-82e7-8f1edb47358c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 04:50:30.724570 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:30.724492    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:30.724570 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:30.724530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:30.724793 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.724636    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlb4b" podUID="f3fb9c3b-3508-423f-82e7-8f1edb47358c"
Jul 17 04:50:30.724863 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:30.724796    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nwkj" podUID="cc964d46-6df9-4848-8b80-ce5423eab530"
Jul 17 04:50:31.565837 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.565808    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-139.ec2.internal" event="NodeReady"
Jul 17 04:50:31.566357 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.565926    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 04:50:31.634384 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.634347    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f6cd5d478-8cfdf"]
Jul 17 04:50:31.636167 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.636151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.639048 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.639022    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 04:50:31.639157 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.639087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bbzx9\""
Jul 17 04:50:31.639157 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.639029    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 04:50:31.639157 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.639124    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 04:50:31.645717 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.644990    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 04:50:31.646210 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.646193    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4pnz8"]
Jul 17 04:50:31.647902 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.647887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.648064 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.648047    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qmwff"]
Jul 17 04:50:31.649731 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.649715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.650612 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.650571    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 04:50:31.650833 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.650670    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 04:50:31.650833 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.650754    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kswzc\""
Jul 17 04:50:31.651571 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.651556    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6cd5d478-8cfdf"]
Jul 17 04:50:31.652750 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.652554    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 04:50:31.652750 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.652568    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 04:50:31.652750 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.652667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 04:50:31.652750 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.652719    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-swxjd\""
Jul 17 04:50:31.682282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.671098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4pnz8"]
Jul 17 04:50:31.682282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.671166    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmwff"]
Jul 17 04:50:31.682282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.671193    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tbzk2"]
Jul 17 04:50:31.691282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.683992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.691282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.687335    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbzk2"]
Jul 17 04:50:31.691282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.688468    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdz6x\""
Jul 17 04:50:31.691282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.688749    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 04:50:31.691282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.689012    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 04:50:31.691282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.689181    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 04:50:31.696436 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.696095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 04:50:31.724465 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.724437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:50:31.726785 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.726765    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 04:50:31.812428 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8vh\" (UniqueName: \"kubernetes.io/projected/642d9a54-09c6-4827-9500-18ae6b91a4e5-kube-api-access-lg8vh\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.812428 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db937e1e-b076-4752-8f84-4e7704ace47f-cert\") pod \"ingress-canary-qmwff\" (UID: \"db937e1e-b076-4752-8f84-4e7704ace47f\") " pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.812623 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642d9a54-09c6-4827-9500-18ae6b91a4e5-tmp-dir\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.812623 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.812623 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xls\" (UniqueName: \"kubernetes.io/projected/db937e1e-b076-4752-8f84-4e7704ace47f-kube-api-access-65xls\") pod \"ingress-canary-qmwff\" (UID: \"db937e1e-b076-4752-8f84-4e7704ace47f\") " pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.812623 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-trusted-ca\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812623 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-bound-sa-token\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812763 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-image-registry-private-configuration\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812763 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aed5dbbc-7e65-4ddb-8512-a013594b38eb-ca-trust-extracted\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812763 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.812763 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jz58\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-kube-api-access-4jz58\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812763 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/642d9a54-09c6-4827-9500-18ae6b91a4e5-config-volume\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.812894 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-certificates\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812894 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tplc\" (UniqueName: \"kubernetes.io/projected/40def634-4715-4391-ad45-4c6acb03ce21-kube-api-access-2tplc\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.812894 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40def634-4715-4391-ad45-4c6acb03ce21-data-volume\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.812894 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.812894 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-installation-pull-secrets\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.813025 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40def634-4715-4391-ad45-4c6acb03ce21-crio-socket\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.813025 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.812953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.911516 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.911436    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6552ab-867d-4bc6-8958-d12371149d7d" containerID="3e7f6ae85a69b472c78d9c628dc18a79297cdb5361648cbd9e6a3e2bb6fc80c7" exitCode=0
Jul 17 04:50:31.911516 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.911492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerDied","Data":"3e7f6ae85a69b472c78d9c628dc18a79297cdb5361648cbd9e6a3e2bb6fc80c7"}
Jul 17 04:50:31.913305 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.913363 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xls\" (UniqueName: \"kubernetes.io/projected/db937e1e-b076-4752-8f84-4e7704ace47f-kube-api-access-65xls\") pod \"ingress-canary-qmwff\" (UID: \"db937e1e-b076-4752-8f84-4e7704ace47f\") " pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.913363 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-trusted-ca\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.913363 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-bound-sa-token\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.913517 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-image-registry-private-configuration\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.913517 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.913484    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 04:50:31.913598 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.913562    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls podName:642d9a54-09c6-4827-9500-18ae6b91a4e5 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:32.413539776 +0000 UTC m=+36.150602103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls") pod "dns-default-4pnz8" (UID: "642d9a54-09c6-4827-9500-18ae6b91a4e5") : secret "dns-default-metrics-tls" not found
Jul 17 04:50:31.913662 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aed5dbbc-7e65-4ddb-8512-a013594b38eb-ca-trust-extracted\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.913662 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.913662 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jz58\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-kube-api-access-4jz58\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.913786 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/642d9a54-09c6-4827-9500-18ae6b91a4e5-config-volume\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.913786 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-certificates\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.913876 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.913859    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:31.913929 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.913914    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:32.413896296 +0000 UTC m=+36.150958613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:31.913977 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tplc\" (UniqueName: \"kubernetes.io/projected/40def634-4715-4391-ad45-4c6acb03ce21-kube-api-access-2tplc\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.914018 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aed5dbbc-7e65-4ddb-8512-a013594b38eb-ca-trust-extracted\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.914018 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.913991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40def634-4715-4391-ad45-4c6acb03ce21-data-volume\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.914104 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.914104 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-installation-pull-secrets\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.914104 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40def634-4715-4391-ad45-4c6acb03ce21-crio-socket\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.914226 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-certificates\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.914226 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.914226 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8vh\" (UniqueName: \"kubernetes.io/projected/642d9a54-09c6-4827-9500-18ae6b91a4e5-kube-api-access-lg8vh\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.914226 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.914195    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 04:50:31.914226 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db937e1e-b076-4752-8f84-4e7704ace47f-cert\") pod \"ingress-canary-qmwff\" (UID: \"db937e1e-b076-4752-8f84-4e7704ace47f\") " pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.914479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40def634-4715-4391-ad45-4c6acb03ce21-data-volume\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.914479 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.914276    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:32.414237175 +0000 UTC m=+36.151299490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : configmap "kube-rbac-proxy" not found
Jul 17 04:50:31.914479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642d9a54-09c6-4827-9500-18ae6b91a4e5-tmp-dir\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.914479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914466    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-trusted-ca\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.914479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/642d9a54-09c6-4827-9500-18ae6b91a4e5-config-volume\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.914708 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.914544    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 04:50:31.914708 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.914557    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6cd5d478-8cfdf: secret "image-registry-tls" not found
Jul 17 04:50:31.914708 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:31.914592    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls podName:aed5dbbc-7e65-4ddb-8512-a013594b38eb nodeName:}" failed. No retries permitted until 2026-07-17 04:50:32.414579242 +0000 UTC m=+36.151641561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls") pod "image-registry-5f6cd5d478-8cfdf" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb") : secret "image-registry-tls" not found
Jul 17 04:50:31.914708 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642d9a54-09c6-4827-9500-18ae6b91a4e5-tmp-dir\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.914708 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.914662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40def634-4715-4391-ad45-4c6acb03ce21-crio-socket\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.918078 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.918049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-installation-pull-secrets\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.918167 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.918083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-image-registry-private-configuration\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.918207 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.918188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db937e1e-b076-4752-8f84-4e7704ace47f-cert\") pod \"ingress-canary-qmwff\" (UID: \"db937e1e-b076-4752-8f84-4e7704ace47f\") " pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.925331 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.925304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8vh\" (UniqueName: \"kubernetes.io/projected/642d9a54-09c6-4827-9500-18ae6b91a4e5-kube-api-access-lg8vh\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:31.925479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.925458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xls\" (UniqueName: \"kubernetes.io/projected/db937e1e-b076-4752-8f84-4e7704ace47f-kube-api-access-65xls\") pod \"ingress-canary-qmwff\" (UID: \"db937e1e-b076-4752-8f84-4e7704ace47f\") " pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:31.925696 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.925667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jz58\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-kube-api-access-4jz58\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.925991 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.925968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tplc\" (UniqueName: \"kubernetes.io/projected/40def634-4715-4391-ad45-4c6acb03ce21-kube-api-access-2tplc\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:31.926649 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.926627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-bound-sa-token\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:31.962893 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:31.962865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmwff"
Jul 17 04:50:32.100962 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.100780    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmwff"]
Jul 17 04:50:32.104109 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:32.104078    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb937e1e_b076_4752_8f84_4e7704ace47f.slice/crio-97db10cfe332280050a4d13679912356a7eb7d48e5ed329e23e327247f3fc28d WatchSource:0}: Error finding container 97db10cfe332280050a4d13679912356a7eb7d48e5ed329e23e327247f3fc28d: Status 404 returned error can't find the container with id 97db10cfe332280050a4d13679912356a7eb7d48e5ed329e23e327247f3fc28d
Jul 17 04:50:32.267541 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.267510    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"]
Jul 17 04:50:32.269496 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.269480    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"
Jul 17 04:50:32.272458 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.272439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gjhmn\""
Jul 17 04:50:32.272759 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.272739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 04:50:32.272850 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.272828    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 04:50:32.280780 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.280758    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"]
Jul 17 04:50:32.418277 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.418230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:32.418439 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.418325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:32.418439 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.418367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:32.418439 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418376    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 04:50:32.418439 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418395    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6cd5d478-8cfdf: secret "image-registry-tls" not found
Jul 17 04:50:32.418439 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418441    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls podName:aed5dbbc-7e65-4ddb-8512-a013594b38eb nodeName:}" failed. No retries permitted until 2026-07-17 04:50:33.41842705 +0000 UTC m=+37.155489364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls") pod "image-registry-5f6cd5d478-8cfdf" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb") : secret "image-registry-tls" not found
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418444    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418481    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418495    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:33.418482948 +0000 UTC m=+37.155545262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : configmap "kube-rbac-proxy" not found
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.418523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418541    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls podName:642d9a54-09c6-4827-9500-18ae6b91a4e5 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:33.418527218 +0000 UTC m=+37.155589548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls") pod "dns-default-4pnz8" (UID: "642d9a54-09c6-4827-9500-18ae6b91a4e5") : secret "dns-default-metrics-tls" not found
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.418572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpqs\" (UniqueName: \"kubernetes.io/projected/8ee5eef8-1714-4f08-8280-eaa0f6839005-kube-api-access-svpqs\") pod \"migrator-7cc9c7ffd7-vrhtt\" (UID: \"8ee5eef8-1714-4f08-8280-eaa0f6839005\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418593    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:32.418706 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:32.418630    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:33.418621755 +0000 UTC m=+37.155684069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:32.519833 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.519739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpqs\" (UniqueName: \"kubernetes.io/projected/8ee5eef8-1714-4f08-8280-eaa0f6839005-kube-api-access-svpqs\") pod \"migrator-7cc9c7ffd7-vrhtt\" (UID: \"8ee5eef8-1714-4f08-8280-eaa0f6839005\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"
Jul 17 04:50:32.529311 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.529281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpqs\" (UniqueName: \"kubernetes.io/projected/8ee5eef8-1714-4f08-8280-eaa0f6839005-kube-api-access-svpqs\") pod \"migrator-7cc9c7ffd7-vrhtt\" (UID: \"8ee5eef8-1714-4f08-8280-eaa0f6839005\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"
Jul 17 04:50:32.578569 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.578534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"
Jul 17 04:50:32.707542 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.707508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt"]
Jul 17 04:50:32.711206 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:32.711175    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee5eef8_1714_4f08_8280_eaa0f6839005.slice/crio-60b1609eb3881c3b7cd116858d5e7adab53243d3c7cb92a9db1fbce7db6b0fe6 WatchSource:0}: Error finding container 60b1609eb3881c3b7cd116858d5e7adab53243d3c7cb92a9db1fbce7db6b0fe6: Status 404 returned error can't find the container with id 60b1609eb3881c3b7cd116858d5e7adab53243d3c7cb92a9db1fbce7db6b0fe6
Jul 17 04:50:32.728533 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.728513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:50:32.728730 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.728704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:50:32.731045 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.731025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 04:50:32.731142 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.731049    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 04:50:32.731142 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.731129    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 04:50:32.731228 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.731153    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zwbtr\""
Jul 17 04:50:32.731228 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.731210    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mhn7j\""
Jul 17 04:50:32.915445 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.915357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt" event={"ID":"8ee5eef8-1714-4f08-8280-eaa0f6839005","Type":"ContainerStarted","Data":"60b1609eb3881c3b7cd116858d5e7adab53243d3c7cb92a9db1fbce7db6b0fe6"}
Jul 17 04:50:32.919055 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.919027    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c6552ab-867d-4bc6-8958-d12371149d7d" containerID="a680057ee6a949c7f04ef6c33510597227640042a6732d8e7a11e910e666cb5d" exitCode=0
Jul 17 04:50:32.919192 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.919114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerDied","Data":"a680057ee6a949c7f04ef6c33510597227640042a6732d8e7a11e910e666cb5d"}
Jul 17 04:50:32.920236 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.920214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmwff" event={"ID":"db937e1e-b076-4752-8f84-4e7704ace47f","Type":"ContainerStarted","Data":"97db10cfe332280050a4d13679912356a7eb7d48e5ed329e23e327247f3fc28d"}
Jul 17 04:50:32.953936 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:32.953910    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6s8kt_90ebd0bd-0591-47a7-b42f-79c949d53fe7/dns-node-resolver/0.log"
Jul 17 04:50:33.427178 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.427140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.427207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.427279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427298    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.427330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427335    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427357    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6cd5d478-8cfdf: secret "image-registry-tls" not found
Jul 17 04:50:33.427365 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427363    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:35.427343378 +0000 UTC m=+39.164405713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : configmap "kube-rbac-proxy" not found
Jul 17 04:50:33.427671 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427405    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:33.427671 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427411    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls podName:aed5dbbc-7e65-4ddb-8512-a013594b38eb nodeName:}" failed. No retries permitted until 2026-07-17 04:50:35.427391487 +0000 UTC m=+39.164453804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls") pod "image-registry-5f6cd5d478-8cfdf" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb") : secret "image-registry-tls" not found
Jul 17 04:50:33.427671 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427438    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:35.42742819 +0000 UTC m=+39.164490504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:33.427671 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427491    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 04:50:33.427671 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:33.427518    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls podName:642d9a54-09c6-4827-9500-18ae6b91a4e5 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:35.427508482 +0000 UTC m=+39.164570803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls") pod "dns-default-4pnz8" (UID: "642d9a54-09c6-4827-9500-18ae6b91a4e5") : secret "dns-default-metrics-tls" not found
Jul 17 04:50:33.670209 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.670173    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tbx5w"]
Jul 17 04:50:33.674986 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.674962    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.678021 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.677951    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 04:50:33.678153 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.677953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 04:50:33.679063 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.679042    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 04:50:33.679252 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.679111    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 04:50:33.679252 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.679042    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7lt8p\""
Jul 17 04:50:33.680631 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.680582    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tbx5w"]
Jul 17 04:50:33.831144 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.830928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/773da5ca-afb4-4327-b10f-36757d25810f-signing-key\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.831326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.831153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/773da5ca-afb4-4327-b10f-36757d25810f-signing-cabundle\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.831326 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.831217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw79\" (UniqueName: \"kubernetes.io/projected/773da5ca-afb4-4327-b10f-36757d25810f-kube-api-access-9rw79\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.927176 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.927139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d282m" event={"ID":"2c6552ab-867d-4bc6-8958-d12371149d7d","Type":"ContainerStarted","Data":"d1f028bd9f0e4df05d7c047b8457ab1b3a9e9cbdcd93329301cdf36bc3819c6f"}
Jul 17 04:50:33.931825 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.931753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/773da5ca-afb4-4327-b10f-36757d25810f-signing-key\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.931825 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.931790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/773da5ca-afb4-4327-b10f-36757d25810f-signing-cabundle\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.931996 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.931848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw79\" (UniqueName: \"kubernetes.io/projected/773da5ca-afb4-4327-b10f-36757d25810f-kube-api-access-9rw79\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.932891 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.932865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/773da5ca-afb4-4327-b10f-36757d25810f-signing-cabundle\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.934937 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.934918    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmd9s_3efb5631-c960-43ec-86b8-3038e6486013/node-ca/0.log"
Jul 17 04:50:33.936004 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.935979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/773da5ca-afb4-4327-b10f-36757d25810f-signing-key\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.939951 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.939930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw79\" (UniqueName: \"kubernetes.io/projected/773da5ca-afb4-4327-b10f-36757d25810f-kube-api-access-9rw79\") pod \"service-ca-58598695f7-tbx5w\" (UID: \"773da5ca-afb4-4327-b10f-36757d25810f\") " pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:33.950547 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.950222    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d282m" podStartSLOduration=8.446272245 podStartE2EDuration="37.950202831s" totalImagesPullingTime="29.503930586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:33.948540943 +0000 UTC m=+37.685603329" watchObservedRunningTime="2026-07-17 04:50:33.950202831 +0000 UTC m=+37.687265168"
Jul 17 04:50:33.990837 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:33.990800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-tbx5w"
Jul 17 04:50:35.189866 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.189827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tbx5w"]
Jul 17 04:50:35.193942 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:35.193916    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773da5ca_afb4_4327_b10f_36757d25810f.slice/crio-c626b8002ab7e8e53ed51397abd64484a188dcbedc958f520f615634d3065210 WatchSource:0}: Error finding container c626b8002ab7e8e53ed51397abd64484a188dcbedc958f520f615634d3065210: Status 404 returned error can't find the container with id c626b8002ab7e8e53ed51397abd64484a188dcbedc958f520f615634d3065210
Jul 17 04:50:35.446305 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.446054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:35.446477 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.446349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:35.446477 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446213    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:35.446477 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.446401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:35.446477 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:39.446424226 +0000 UTC m=+43.183486544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:35.446477 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446475    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.446482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446504    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446531    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6cd5d478-8cfdf: secret "image-registry-tls" not found
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446545    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446512    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:39.446500233 +0000 UTC m=+43.183562561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : configmap "kube-rbac-proxy" not found
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446588    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls podName:aed5dbbc-7e65-4ddb-8512-a013594b38eb nodeName:}" failed. No retries permitted until 2026-07-17 04:50:39.446576465 +0000 UTC m=+43.183638778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls") pod "image-registry-5f6cd5d478-8cfdf" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb") : secret "image-registry-tls" not found
Jul 17 04:50:35.446696 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:35.446638    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls podName:642d9a54-09c6-4827-9500-18ae6b91a4e5 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:39.4466055 +0000 UTC m=+43.183667832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls") pod "dns-default-4pnz8" (UID: "642d9a54-09c6-4827-9500-18ae6b91a4e5") : secret "dns-default-metrics-tls" not found
Jul 17 04:50:35.933968 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.933926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmwff" event={"ID":"db937e1e-b076-4752-8f84-4e7704ace47f","Type":"ContainerStarted","Data":"024be20283363ba49ebaa63455008df42cb914c7209caa35f26de96ca7092947"}
Jul 17 04:50:35.936080 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.936049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt" event={"ID":"8ee5eef8-1714-4f08-8280-eaa0f6839005","Type":"ContainerStarted","Data":"3170c30a7316091afbdc4e149ed8383b8e325505f06567519ce562bd9906f92e"}
Jul 17 04:50:35.936200 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.936111    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt" event={"ID":"8ee5eef8-1714-4f08-8280-eaa0f6839005","Type":"ContainerStarted","Data":"cbcad1daf8e30ec3143c388d5e23a36662839654e7dd6b559986719c4d1deab4"}
Jul 17 04:50:35.937536 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.937499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-tbx5w" event={"ID":"773da5ca-afb4-4327-b10f-36757d25810f","Type":"ContainerStarted","Data":"c626b8002ab7e8e53ed51397abd64484a188dcbedc958f520f615634d3065210"}
Jul 17 04:50:35.948676 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:35.948624    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qmwff" podStartSLOduration=1.990534368 podStartE2EDuration="4.948605659s" totalImagesPullingTime="2.958071291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:35.947518152 +0000 UTC m=+39.684580489" watchObservedRunningTime="2026-07-17 04:50:35.948605659 +0000 UTC m=+39.685667996"
Jul 17 04:50:37.944408 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:37.944376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-tbx5w" event={"ID":"773da5ca-afb4-4327-b10f-36757d25810f","Type":"ContainerStarted","Data":"e60ecf1626d73e1b724fe45fa2b6b98dd8c26fb908d503e6ea87c1bed912bb8d"}
Jul 17 04:50:37.959156 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:37.959106    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vrhtt" podStartSLOduration=3.606930081 podStartE2EDuration="5.959084436s" totalImagesPullingTime="2.352154355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:35.960870209 +0000 UTC m=+39.697932566" watchObservedRunningTime="2026-07-17 04:50:37.959084436 +0000 UTC m=+41.696146770"
Jul 17 04:50:39.482124 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:39.482080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:39.482169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:39.482215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482254    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482302    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6cd5d478-8cfdf: secret "image-registry-tls" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482317    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482366    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls podName:aed5dbbc-7e65-4ddb-8512-a013594b38eb nodeName:}" failed. No retries permitted until 2026-07-17 04:50:47.482346646 +0000 UTC m=+51.219408959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls") pod "image-registry-5f6cd5d478-8cfdf" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb") : secret "image-registry-tls" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482366    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482387    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:47.482376469 +0000 UTC m=+51.219438785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : configmap "kube-rbac-proxy" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:39.482283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482396    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482416    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:47.482400854 +0000 UTC m=+51.219463171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:39.482592 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:39.482462    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls podName:642d9a54-09c6-4827-9500-18ae6b91a4e5 nodeName:}" failed. No retries permitted until 2026-07-17 04:50:47.482446654 +0000 UTC m=+51.219508985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls") pod "dns-default-4pnz8" (UID: "642d9a54-09c6-4827-9500-18ae6b91a4e5") : secret "dns-default-metrics-tls" not found
Jul 17 04:50:47.544632 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.544596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:47.545127 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.544649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:47.545127 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.544700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:47.545127 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.544749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:47.545127 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:47.544849    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:47.545127 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:50:47.544945    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls podName:40def634-4715-4391-ad45-4c6acb03ce21 nodeName:}" failed. No retries permitted until 2026-07-17 04:51:03.544923849 +0000 UTC m=+67.281986168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbzk2" (UID: "40def634-4715-4391-ad45-4c6acb03ce21") : secret "insights-runtime-extractor-tls" not found
Jul 17 04:50:47.545371 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.545281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40def634-4715-4391-ad45-4c6acb03ce21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:50:47.548712 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.548685    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/642d9a54-09c6-4827-9500-18ae6b91a4e5-metrics-tls\") pod \"dns-default-4pnz8\" (UID: \"642d9a54-09c6-4827-9500-18ae6b91a4e5\") " pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:47.548815 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.548695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"image-registry-5f6cd5d478-8cfdf\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") " pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:47.557345 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.557319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:47.702829 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.702694    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-tbx5w" podStartSLOduration=12.341746375 podStartE2EDuration="14.702675893s" totalImagesPullingTime="2.360929518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:37.959295569 +0000 UTC m=+41.696357930" watchObservedRunningTime="2026-07-17 04:50:47.702675893 +0000 UTC m=+51.439738302"
Jul 17 04:50:47.702890 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.702873    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4pnz8"]
Jul 17 04:50:47.705341 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:47.705311    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642d9a54_09c6_4827_9500_18ae6b91a4e5.slice/crio-e758e5de6bbaacecbf05395e11c28e228f646a2d18ba4174c24c806c4b4e0d0a WatchSource:0}: Error finding container e758e5de6bbaacecbf05395e11c28e228f646a2d18ba4174c24c806c4b4e0d0a: Status 404 returned error can't find the container with id e758e5de6bbaacecbf05395e11c28e228f646a2d18ba4174c24c806c4b4e0d0a
Jul 17 04:50:47.847251 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.847157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:47.972320 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.972251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4pnz8" event={"ID":"642d9a54-09c6-4827-9500-18ae6b91a4e5","Type":"ContainerStarted","Data":"e758e5de6bbaacecbf05395e11c28e228f646a2d18ba4174c24c806c4b4e0d0a"}
Jul 17 04:50:47.996344 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:47.996314    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6cd5d478-8cfdf"]
Jul 17 04:50:47.998958 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:47.998925    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed5dbbc_7e65_4ddb_8512_a013594b38eb.slice/crio-4fefcb357a7482cb5f07306b98a46d06cdc17e7d595b0d6f1f9207bc8576ab7e WatchSource:0}: Error finding container 4fefcb357a7482cb5f07306b98a46d06cdc17e7d595b0d6f1f9207bc8576ab7e: Status 404 returned error can't find the container with id 4fefcb357a7482cb5f07306b98a46d06cdc17e7d595b0d6f1f9207bc8576ab7e
Jul 17 04:50:48.977054 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:48.977017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" event={"ID":"aed5dbbc-7e65-4ddb-8512-a013594b38eb","Type":"ContainerStarted","Data":"9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d"}
Jul 17 04:50:48.977633 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:48.977074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" event={"ID":"aed5dbbc-7e65-4ddb-8512-a013594b38eb","Type":"ContainerStarted","Data":"4fefcb357a7482cb5f07306b98a46d06cdc17e7d595b0d6f1f9207bc8576ab7e"}
Jul 17 04:50:48.977633 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:48.977163    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:50:48.996972 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:48.996924    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" podStartSLOduration=24.996908398 podStartE2EDuration="24.996908398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:48.996137161 +0000 UTC m=+52.733199517" watchObservedRunningTime="2026-07-17 04:50:48.996908398 +0000 UTC m=+52.733970733"
Jul 17 04:50:49.982433 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:49.982406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4pnz8" event={"ID":"642d9a54-09c6-4827-9500-18ae6b91a4e5","Type":"ContainerStarted","Data":"cace0f48fe45fc28d7fb1df2462bf6c8dbc63ae17fc5a1ad55226c7b3a118b3f"}
Jul 17 04:50:50.986460 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:50.986416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4pnz8" event={"ID":"642d9a54-09c6-4827-9500-18ae6b91a4e5","Type":"ContainerStarted","Data":"9c0d10f30750857f3869fab121a57d6e708225106ea25c243e65b76383f3b706"}
Jul 17 04:50:50.986839 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:50.986553    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:50:51.005824 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:51.005776    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4pnz8" podStartSLOduration=17.901444569 podStartE2EDuration="20.005758514s" totalImagesPullingTime="2.104313945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:51.004005137 +0000 UTC m=+54.741067472" watchObservedRunningTime="2026-07-17 04:50:51.005758514 +0000 UTC m=+54.742820851"
Jul 17 04:50:56.371824 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.371790    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"]
Jul 17 04:50:56.374617 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.374596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.378577 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.378548    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 04:50:56.378867 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.378848    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 04:50:56.379081 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.379065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 04:50:56.379142 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.379076    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 04:50:56.379357 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.379339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s6js4\""
Jul 17 04:50:56.380189 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.380167    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"]
Jul 17 04:50:56.382420 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.382391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.384801 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.384781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 04:50:56.384885 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.384806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 04:50:56.386207 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.386194    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 04:50:56.386249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.386212    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 04:50:56.392807 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.392784    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"]
Jul 17 04:50:56.396679 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.396655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6cd5d478-8cfdf"]
Jul 17 04:50:56.405868 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.405843    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"]
Jul 17 04:50:56.411924 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.411885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93600ef3-8421-431b-968e-69f70013de77-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6\" (UID: \"93600ef3-8421-431b-968e-69f70013de77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.412086 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.411933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.412086 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.411991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-ca\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.412086 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.412014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb45daba-2cae-49e6-94f5-8e1547e73d16-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.412249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.412098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xv6c\" (UniqueName: \"kubernetes.io/projected/cb45daba-2cae-49e6-94f5-8e1547e73d16-kube-api-access-2xv6c\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.412249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.412127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.412249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.412152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtc5h\" (UniqueName: \"kubernetes.io/projected/93600ef3-8421-431b-968e-69f70013de77-kube-api-access-wtc5h\") pod \"managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6\" (UID: \"93600ef3-8421-431b-968e-69f70013de77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.412249 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.412178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-hub\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.471759 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.471727    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8677c6c9b4-wrjqg"]
Jul 17 04:50:56.473881 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.473865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.490531 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.490476    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8677c6c9b4-wrjqg"]
Jul 17 04:50:56.512893 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.512855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93600ef3-8421-431b-968e-69f70013de77-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6\" (UID: \"93600ef3-8421-431b-968e-69f70013de77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.512893 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.512891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.513129 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.512931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-ca\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.513129 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.512963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec735f34-6ae1-41af-ba35-692446d958f2-ca-trust-extracted\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513129 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.512983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb45daba-2cae-49e6-94f5-8e1547e73d16-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.513129 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xv6c\" (UniqueName: \"kubernetes.io/projected/cb45daba-2cae-49e6-94f5-8e1547e73d16-kube-api-access-2xv6c\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.513129 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc5h\" (UniqueName: \"kubernetes.io/projected/93600ef3-8421-431b-968e-69f70013de77-kube-api-access-wtc5h\") pod \"managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6\" (UID: \"93600ef3-8421-431b-968e-69f70013de77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec735f34-6ae1-41af-ba35-692446d958f2-image-registry-private-configuration\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec735f34-6ae1-41af-ba35-692446d958f2-trusted-ca\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-hub\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec735f34-6ae1-41af-ba35-692446d958f2-registry-certificates\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-bound-sa-token\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-registry-tls\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec735f34-6ae1-41af-ba35-692446d958f2-installation-pull-secrets\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.513556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gd6\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-kube-api-access-k5gd6\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.514182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.513864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb45daba-2cae-49e6-94f5-8e1547e73d16-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.515784 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.515757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-ca\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.515927 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.515913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-hub\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.516203 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.516179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.516313 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.516296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93600ef3-8421-431b-968e-69f70013de77-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6\" (UID: \"93600ef3-8421-431b-968e-69f70013de77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.516882 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.516862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb45daba-2cae-49e6-94f5-8e1547e73d16-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.537073 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.536945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xv6c\" (UniqueName: \"kubernetes.io/projected/cb45daba-2cae-49e6-94f5-8e1547e73d16-kube-api-access-2xv6c\") pod \"cluster-proxy-proxy-agent-fb8db98fb-j4msk\" (UID: \"cb45daba-2cae-49e6-94f5-8e1547e73d16\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.537445 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.537396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc5h\" (UniqueName: \"kubernetes.io/projected/93600ef3-8421-431b-968e-69f70013de77-kube-api-access-wtc5h\") pod \"managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6\" (UID: \"93600ef3-8421-431b-968e-69f70013de77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.614052 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec735f34-6ae1-41af-ba35-692446d958f2-ca-trust-extracted\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec735f34-6ae1-41af-ba35-692446d958f2-image-registry-private-configuration\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec735f34-6ae1-41af-ba35-692446d958f2-trusted-ca\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec735f34-6ae1-41af-ba35-692446d958f2-registry-certificates\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-bound-sa-token\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-registry-tls\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec735f34-6ae1-41af-ba35-692446d958f2-installation-pull-secrets\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614621 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gd6\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-kube-api-access-k5gd6\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.614621 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.614544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec735f34-6ae1-41af-ba35-692446d958f2-ca-trust-extracted\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.615114 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.615056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec735f34-6ae1-41af-ba35-692446d958f2-registry-certificates\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.615114 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.615059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec735f34-6ae1-41af-ba35-692446d958f2-trusted-ca\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.616968 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.616941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec735f34-6ae1-41af-ba35-692446d958f2-installation-pull-secrets\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.617071 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.617016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec735f34-6ae1-41af-ba35-692446d958f2-image-registry-private-configuration\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.617303 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.617285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-registry-tls\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.625152 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.625096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-bound-sa-token\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.625357 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.625339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gd6\" (UniqueName: \"kubernetes.io/projected/ec735f34-6ae1-41af-ba35-692446d958f2-kube-api-access-k5gd6\") pod \"image-registry-8677c6c9b4-wrjqg\" (UID: \"ec735f34-6ae1-41af-ba35-692446d958f2\") " pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.698934 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.698907    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 04:50:56.705116 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.705093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"
Jul 17 04:50:56.709746 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.709727    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s6js4\""
Jul 17 04:50:56.718072 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.718052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"
Jul 17 04:50:56.796109 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.795532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:56.855189 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.855143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk"]
Jul 17 04:50:56.860051 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:56.860018    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb45daba_2cae_49e6_94f5_8e1547e73d16.slice/crio-3f47856bc91d93bcd30577f7fa2cb07f256269fd0322be8950b6b1dd0945bc77 WatchSource:0}: Error finding container 3f47856bc91d93bcd30577f7fa2cb07f256269fd0322be8950b6b1dd0945bc77: Status 404 returned error can't find the container with id 3f47856bc91d93bcd30577f7fa2cb07f256269fd0322be8950b6b1dd0945bc77
Jul 17 04:50:56.872498 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.872467    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6"]
Jul 17 04:50:56.875821 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:56.875751    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93600ef3_8421_431b_968e_69f70013de77.slice/crio-1735c8b7f2649633b7e60b48858e078c0650b4a12b55522b34a8113c20ca68c0 WatchSource:0}: Error finding container 1735c8b7f2649633b7e60b48858e078c0650b4a12b55522b34a8113c20ca68c0: Status 404 returned error can't find the container with id 1735c8b7f2649633b7e60b48858e078c0650b4a12b55522b34a8113c20ca68c0
Jul 17 04:50:56.910534 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.910506    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nxz9q"
Jul 17 04:50:56.936168 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:56.936136    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8677c6c9b4-wrjqg"]
Jul 17 04:50:56.940098 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:50:56.940074    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec735f34_6ae1_41af_ba35_692446d958f2.slice/crio-355dece94c99bed9e7798683508ae8921af312b63676a0e73312026c077b4409 WatchSource:0}: Error finding container 355dece94c99bed9e7798683508ae8921af312b63676a0e73312026c077b4409: Status 404 returned error can't find the container with id 355dece94c99bed9e7798683508ae8921af312b63676a0e73312026c077b4409
Jul 17 04:50:57.011057 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:57.011026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6" event={"ID":"93600ef3-8421-431b-968e-69f70013de77","Type":"ContainerStarted","Data":"1735c8b7f2649633b7e60b48858e078c0650b4a12b55522b34a8113c20ca68c0"}
Jul 17 04:50:57.012645 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:57.012611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg" event={"ID":"ec735f34-6ae1-41af-ba35-692446d958f2","Type":"ContainerStarted","Data":"37458fed98974265c77d3e0489ab2ced82a9539637d7db87347b7b62e0fc31ca"}
Jul 17 04:50:57.012747 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:57.012700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg" event={"ID":"ec735f34-6ae1-41af-ba35-692446d958f2","Type":"ContainerStarted","Data":"355dece94c99bed9e7798683508ae8921af312b63676a0e73312026c077b4409"}
Jul 17 04:50:57.013776 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:57.013752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk" event={"ID":"cb45daba-2cae-49e6-94f5-8e1547e73d16","Type":"ContainerStarted","Data":"3f47856bc91d93bcd30577f7fa2cb07f256269fd0322be8950b6b1dd0945bc77"}
Jul 17 04:50:58.017147 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:58.017115    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:50:58.041307 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:50:58.041220    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg" podStartSLOduration=2.041198209 podStartE2EDuration="2.041198209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:50:58.040548442 +0000 UTC m=+61.777610780" watchObservedRunningTime="2026-07-17 04:50:58.041198209 +0000 UTC m=+61.778260547"
Jul 17 04:51:00.997986 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:00.997953    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4pnz8"
Jul 17 04:51:01.036527 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:01.036490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk" event={"ID":"cb45daba-2cae-49e6-94f5-8e1547e73d16","Type":"ContainerStarted","Data":"d99119096c3c77426b9f1dba689f067f90bb9bd6734150e02828802f75ce0c2b"}
Jul 17 04:51:01.036717 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:01.036536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk" event={"ID":"cb45daba-2cae-49e6-94f5-8e1547e73d16","Type":"ContainerStarted","Data":"16b4014caf0ee8c8bbea9a2e92276b8db81ddcbabc04f623a189e088eeb78b04"}
Jul 17 04:51:01.036717 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:01.036552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk" event={"ID":"cb45daba-2cae-49e6-94f5-8e1547e73d16","Type":"ContainerStarted","Data":"e9c7eeb4295f5cf819f926cb5ba6b564f2c69961bd04c4a0ad4c1caa9c5d1714"}
Jul 17 04:51:01.055395 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:01.055346    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb8db98fb-j4msk" podStartSLOduration=1.259439988 podStartE2EDuration="5.055328813s" totalImagesPullingTime="3.795888825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:01.055215165 +0000 UTC m=+64.792277501" watchObservedRunningTime="2026-07-17 04:51:01.055328813 +0000 UTC m=+64.792391128"
Jul 17 04:51:02.040850 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.040810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6" event={"ID":"93600ef3-8421-431b-968e-69f70013de77","Type":"ContainerStarted","Data":"e0cafa80a54d154ea9f2b1dae0d891fbe7c19be49999822b31c3fc69c157c6f4"}
Jul 17 04:51:02.056049 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.055998    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cb5fd59b6-zpvk6" podStartSLOduration=1.3904410569999999 podStartE2EDuration="6.055980019s" totalImagesPullingTime="4.665538962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:02.055926842 +0000 UTC m=+65.792989179" watchObservedRunningTime="2026-07-17 04:51:02.055980019 +0000 UTC m=+65.793042355"
Jul 17 04:51:02.468320 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.468224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:51:02.471248 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.471225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 04:51:02.481145 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.481126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5be18d5-6b55-42fa-8669-03ac30ab7f33-original-pull-secret\") pod \"global-pull-secret-syncer-drpqq\" (UID: \"b5be18d5-6b55-42fa-8669-03ac30ab7f33\") " pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:51:02.568703 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.568667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:51:02.568882 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.568714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:51:02.571471 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.571455    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 04:51:02.571530 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.571484    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 04:51:02.581835 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.581803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc964d46-6df9-4848-8b80-ce5423eab530-metrics-certs\") pod \"network-metrics-daemon-9nwkj\" (UID: \"cc964d46-6df9-4848-8b80-ce5423eab530\") " pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:51:02.582123 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.582108    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 04:51:02.592676 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.592656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wvk\" (UniqueName: \"kubernetes.io/projected/f3fb9c3b-3508-423f-82e7-8f1edb47358c-kube-api-access-j4wvk\") pod \"network-check-target-wlb4b\" (UID: \"f3fb9c3b-3508-423f-82e7-8f1edb47358c\") " pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:51:02.632820 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.632783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drpqq"
Jul 17 04:51:02.746342 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.746062    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zwbtr\""
Jul 17 04:51:02.755043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.754577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mhn7j\""
Jul 17 04:51:02.755043 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.754809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nwkj"
Jul 17 04:51:02.761524 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.761440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:51:02.783124 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.783094    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drpqq"]
Jul 17 04:51:02.787392 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:51:02.787349    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5be18d5_6b55_42fa_8669_03ac30ab7f33.slice/crio-777fdb1be097e4994aa73fa8cc7919f9446e7aa8d3306b3a2b7c622e1df7050a WatchSource:0}: Error finding container 777fdb1be097e4994aa73fa8cc7919f9446e7aa8d3306b3a2b7c622e1df7050a: Status 404 returned error can't find the container with id 777fdb1be097e4994aa73fa8cc7919f9446e7aa8d3306b3a2b7c622e1df7050a
Jul 17 04:51:02.901493 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.901468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlb4b"]
Jul 17 04:51:02.904243 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:51:02.904217    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fb9c3b_3508_423f_82e7_8f1edb47358c.slice/crio-198e788998d6ba3eafe69f419d6694dbf7484963516358735ad5be6297214a7a WatchSource:0}: Error finding container 198e788998d6ba3eafe69f419d6694dbf7484963516358735ad5be6297214a7a: Status 404 returned error can't find the container with id 198e788998d6ba3eafe69f419d6694dbf7484963516358735ad5be6297214a7a
Jul 17 04:51:02.918737 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:02.918708    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9nwkj"]
Jul 17 04:51:02.921377 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:51:02.921340    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc964d46_6df9_4848_8b80_ce5423eab530.slice/crio-29f5c4278f8337437aab5271c496c48cff6b4b6d4a452c9e3546d0675ef25ca6 WatchSource:0}: Error finding container 29f5c4278f8337437aab5271c496c48cff6b4b6d4a452c9e3546d0675ef25ca6: Status 404 returned error can't find the container with id 29f5c4278f8337437aab5271c496c48cff6b4b6d4a452c9e3546d0675ef25ca6
Jul 17 04:51:03.045626 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.045535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlb4b" event={"ID":"f3fb9c3b-3508-423f-82e7-8f1edb47358c","Type":"ContainerStarted","Data":"198e788998d6ba3eafe69f419d6694dbf7484963516358735ad5be6297214a7a"}
Jul 17 04:51:03.046676 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.046655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drpqq" event={"ID":"b5be18d5-6b55-42fa-8669-03ac30ab7f33","Type":"ContainerStarted","Data":"777fdb1be097e4994aa73fa8cc7919f9446e7aa8d3306b3a2b7c622e1df7050a"}
Jul 17 04:51:03.047725 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.047703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9nwkj" event={"ID":"cc964d46-6df9-4848-8b80-ce5423eab530","Type":"ContainerStarted","Data":"29f5c4278f8337437aab5271c496c48cff6b4b6d4a452c9e3546d0675ef25ca6"}
Jul 17 04:51:03.578653 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.578612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:51:03.581918 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.581854    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40def634-4715-4391-ad45-4c6acb03ce21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbzk2\" (UID: \"40def634-4715-4391-ad45-4c6acb03ce21\") " pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:51:03.808323 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.808097    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdz6x\""
Jul 17 04:51:03.816464 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:03.816286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbzk2"
Jul 17 04:51:04.014458 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:04.014335    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbzk2"]
Jul 17 04:51:04.223492 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:51:04.223458    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40def634_4715_4391_ad45_4c6acb03ce21.slice/crio-4333f910a9469a73b2af3c9de7ee9ef725663bb0f1cd7ed517b36e32a4a862a9 WatchSource:0}: Error finding container 4333f910a9469a73b2af3c9de7ee9ef725663bb0f1cd7ed517b36e32a4a862a9: Status 404 returned error can't find the container with id 4333f910a9469a73b2af3c9de7ee9ef725663bb0f1cd7ed517b36e32a4a862a9
Jul 17 04:51:05.064142 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:05.063994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbzk2" event={"ID":"40def634-4715-4391-ad45-4c6acb03ce21","Type":"ContainerStarted","Data":"4333f910a9469a73b2af3c9de7ee9ef725663bb0f1cd7ed517b36e32a4a862a9"}
Jul 17 04:51:05.067966 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:05.067274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9nwkj" event={"ID":"cc964d46-6df9-4848-8b80-ce5423eab530","Type":"ContainerStarted","Data":"5042993f1169e56a3af1a8f36d2a9ce1eb767ed5632041072b0b5995730980d5"}
Jul 17 04:51:05.067966 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:05.067924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9nwkj" event={"ID":"cc964d46-6df9-4848-8b80-ce5423eab530","Type":"ContainerStarted","Data":"9ad7fa33a4c3d737e33ca4a7f9171cd7f10a1bf6d176b97c4bd40f5005926531"}
Jul 17 04:51:05.085715 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:05.085149    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9nwkj" podStartSLOduration=67.483697587 podStartE2EDuration="1m9.0851273s" totalImagesPullingTime="1.601429713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:05.082510587 +0000 UTC m=+68.819572972" watchObservedRunningTime="2026-07-17 04:51:05.0851273 +0000 UTC m=+68.822189637"
Jul 17 04:51:06.404240 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:06.404192    2582 patch_prober.go:28] interesting pod/image-registry-5f6cd5d478-8cfdf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 04:51:06.404666 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:06.404287    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" podUID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 04:51:11.097010 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:11.096874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlb4b" event={"ID":"f3fb9c3b-3508-423f-82e7-8f1edb47358c","Type":"ContainerStarted","Data":"8c7138139de232fa9e94354391b7fa62a702189b90c02cbb335f2474095b2e9b"}
Jul 17 04:51:11.097010 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:11.096977    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:51:11.099079 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:11.098817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drpqq" event={"ID":"b5be18d5-6b55-42fa-8669-03ac30ab7f33","Type":"ContainerStarted","Data":"3895700e3cea8e17d6cef502815cb25ecd447a7404f1907daaec5943b7ff33b6"}
Jul 17 04:51:11.100080 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:11.100059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbzk2" event={"ID":"40def634-4715-4391-ad45-4c6acb03ce21","Type":"ContainerStarted","Data":"b9e6ab0511fa84156f470b728517cc7cdc098856629f265c8e23d254f470712b"}
Jul 17 04:51:11.113641 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:11.113595    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wlb4b" podStartSLOduration=67.089814483 podStartE2EDuration="1m15.113579973s" totalImagesPullingTime="8.02376549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:11.112823304 +0000 UTC m=+74.849885654" watchObservedRunningTime="2026-07-17 04:51:11.113579973 +0000 UTC m=+74.850642309"
Jul 17 04:51:11.129528 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:11.129470    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-drpqq" podStartSLOduration=66.990870377 podStartE2EDuration="1m15.129451109s" totalImagesPullingTime="8.138580732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:11.128969575 +0000 UTC m=+74.866031913" watchObservedRunningTime="2026-07-17 04:51:11.129451109 +0000 UTC m=+74.866513445"
Jul 17 04:51:15.116504 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:15.116471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbzk2" event={"ID":"40def634-4715-4391-ad45-4c6acb03ce21","Type":"ContainerStarted","Data":"38cbb6d67d78ec40ef5cdb3d53e67eb08b3b118efd1ce6fd2cfa6f3400774cb1"}
Jul 17 04:51:16.402036 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:16.402009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:51:17.098845 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.098653    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ffnhs"]
Jul 17 04:51:17.109392 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.109338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.112556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.112177    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 04:51:17.112556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.112308    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 04:51:17.112556 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.112423    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 04:51:17.113468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.113145    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 04:51:17.113468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.113288    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 04:51:17.113468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.113320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 04:51:17.113468 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.113376    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2j6t8\""
Jul 17 04:51:17.197453 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wrp\" (UniqueName: \"kubernetes.io/projected/61b8012a-a333-4b4e-b963-859b3513e8f2-kube-api-access-m4wrp\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197453 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197681 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197681 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-sys\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197681 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-root\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197796 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b8012a-a333-4b4e-b963-859b3513e8f2-metrics-client-ca\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197796 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-textfile\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197796 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-wtmp\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.197796 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.197786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.298994 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.298953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b8012a-a333-4b4e-b963-859b3513e8f2-metrics-client-ca\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-textfile\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-wtmp\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wrp\" (UniqueName: \"kubernetes.io/projected/61b8012a-a333-4b4e-b963-859b3513e8f2-kube-api-access-m4wrp\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299182 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299524 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-sys\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299524 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-root\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299524 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-root\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.299952 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.299927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b8012a-a333-4b4e-b963-859b3513e8f2-metrics-client-ca\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.300176 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.300154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-textfile\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.300338 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.300321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-wtmp\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.300432 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:51:17.300417    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 04:51:17.300495 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:51:17.300481    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls podName:61b8012a-a333-4b4e-b963-859b3513e8f2 nodeName:}" failed. No retries permitted until 2026-07-17 04:51:17.800460034 +0000 UTC m=+81.537522352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls") pod "node-exporter-ffnhs" (UID: "61b8012a-a333-4b4e-b963-859b3513e8f2") : secret "node-exporter-tls" not found
Jul 17 04:51:17.300964 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.300944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b8012a-a333-4b4e-b963-859b3513e8f2-sys\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.301200 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.301180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.316288 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.303677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.316288 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.309818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wrp\" (UniqueName: \"kubernetes.io/projected/61b8012a-a333-4b4e-b963-859b3513e8f2-kube-api-access-m4wrp\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.804582 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:17.804540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:17.805060 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:51:17.804747    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 04:51:17.805060 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:51:17.804812    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls podName:61b8012a-a333-4b4e-b963-859b3513e8f2 nodeName:}" failed. No retries permitted until 2026-07-17 04:51:18.804791858 +0000 UTC m=+82.541854187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls") pod "node-exporter-ffnhs" (UID: "61b8012a-a333-4b4e-b963-859b3513e8f2") : secret "node-exporter-tls" not found
Jul 17 04:51:18.813683 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:18.813650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:18.816067 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:18.816044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b8012a-a333-4b4e-b963-859b3513e8f2-node-exporter-tls\") pod \"node-exporter-ffnhs\" (UID: \"61b8012a-a333-4b4e-b963-859b3513e8f2\") " pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:18.923436 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:18.923396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffnhs"
Jul 17 04:51:18.935281 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:51:18.935233    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b8012a_a333_4b4e_b963_859b3513e8f2.slice/crio-f59452cfa581b82c5afc5d6e5effb784ee3e1f96551015afa1e0e3465687fa31 WatchSource:0}: Error finding container f59452cfa581b82c5afc5d6e5effb784ee3e1f96551015afa1e0e3465687fa31: Status 404 returned error can't find the container with id f59452cfa581b82c5afc5d6e5effb784ee3e1f96551015afa1e0e3465687fa31
Jul 17 04:51:19.025628 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:19.025601    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8677c6c9b4-wrjqg"
Jul 17 04:51:19.132937 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:19.132851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffnhs" event={"ID":"61b8012a-a333-4b4e-b963-859b3513e8f2","Type":"ContainerStarted","Data":"f59452cfa581b82c5afc5d6e5effb784ee3e1f96551015afa1e0e3465687fa31"}
Jul 17 04:51:19.134813 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:19.134787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbzk2" event={"ID":"40def634-4715-4391-ad45-4c6acb03ce21","Type":"ContainerStarted","Data":"6ee1a2c2a25e67f0e1b5710de1131882f5f459fa50756032d1110e5d8231f030"}
Jul 17 04:51:19.155618 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:19.155501    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tbzk2" podStartSLOduration=34.237944338 podStartE2EDuration="48.15548071s" totalImagesPullingTime="13.917536372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 04:50:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:19.155051956 +0000 UTC m=+82.892114309" watchObservedRunningTime="2026-07-17 04:51:19.15548071 +0000 UTC m=+82.892543046"
Jul 17 04:51:21.144428 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.144387    2582 generic.go:356] "Generic (PLEG): container finished" podID="61b8012a-a333-4b4e-b963-859b3513e8f2" containerID="4fe8de5f8c43ae1296f5ed5d766d79be211a611ad91d81bcba1bb378e7a2e954" exitCode=0
Jul 17 04:51:21.144896 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.144468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffnhs" event={"ID":"61b8012a-a333-4b4e-b963-859b3513e8f2","Type":"ContainerDied","Data":"4fe8de5f8c43ae1296f5ed5d766d79be211a611ad91d81bcba1bb378e7a2e954"}
Jul 17 04:51:21.418188 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.418093    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" podUID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" containerName="registry" containerID="cri-o://9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d" gracePeriod=30
Jul 17 04:51:21.662454 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.662429    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:51:21.735716 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735681    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-installation-pull-secrets\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.735908 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735728    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jz58\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-kube-api-access-4jz58\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.735908 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735757    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-bound-sa-token\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.735908 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735789    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-image-registry-private-configuration\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.735908 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.735908 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735848    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aed5dbbc-7e65-4ddb-8512-a013594b38eb-ca-trust-extracted\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.735908 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-trusted-ca\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.736172 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.735915    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-certificates\") pod \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\" (UID: \"aed5dbbc-7e65-4ddb-8512-a013594b38eb\") "
Jul 17 04:51:21.736696 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.736602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 04:51:21.736696 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.736609    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 04:51:21.738525 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.738492    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-kube-api-access-4jz58" (OuterVolumeSpecName: "kube-api-access-4jz58") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "kube-api-access-4jz58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 04:51:21.738643 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.738598    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 04:51:21.738731 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.738716    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 04:51:21.738778 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.738711    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 04:51:21.738778 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.738749    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 04:51:21.744059 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.744018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed5dbbc-7e65-4ddb-8512-a013594b38eb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "aed5dbbc-7e65-4ddb-8512-a013594b38eb" (UID: "aed5dbbc-7e65-4ddb-8512-a013594b38eb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 04:51:21.836930 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836895    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-image-registry-private-configuration\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.836930 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836929    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-tls\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.837141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836944    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aed5dbbc-7e65-4ddb-8512-a013594b38eb-ca-trust-extracted\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.837141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836955    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-trusted-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.837141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836966    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aed5dbbc-7e65-4ddb-8512-a013594b38eb-registry-certificates\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.837141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836978    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aed5dbbc-7e65-4ddb-8512-a013594b38eb-installation-pull-secrets\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.837141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.836991    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jz58\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-kube-api-access-4jz58\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:21.837141 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:21.837001    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aed5dbbc-7e65-4ddb-8512-a013594b38eb-bound-sa-token\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:51:22.149789 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.149703    2582 generic.go:356] "Generic (PLEG): container finished" podID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" containerID="9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d" exitCode=0
Jul 17 04:51:22.149789 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.149751    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf"
Jul 17 04:51:22.150298 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.149791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" event={"ID":"aed5dbbc-7e65-4ddb-8512-a013594b38eb","Type":"ContainerDied","Data":"9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d"}
Jul 17 04:51:22.150298 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.149846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6cd5d478-8cfdf" event={"ID":"aed5dbbc-7e65-4ddb-8512-a013594b38eb","Type":"ContainerDied","Data":"4fefcb357a7482cb5f07306b98a46d06cdc17e7d595b0d6f1f9207bc8576ab7e"}
Jul 17 04:51:22.150298 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.149873    2582 scope.go:117] "RemoveContainer" containerID="9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d"
Jul 17 04:51:22.152407 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.152383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffnhs" event={"ID":"61b8012a-a333-4b4e-b963-859b3513e8f2","Type":"ContainerStarted","Data":"583a28dc8e697421276728cb08ba19009f4a12751a50311f0dc1f59617b354f5"}
Jul 17 04:51:22.152520 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.152427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffnhs" event={"ID":"61b8012a-a333-4b4e-b963-859b3513e8f2","Type":"ContainerStarted","Data":"0d454fca44b9d844536844ad65be02c4b361da5244568d0917b87adede83c6b8"}
Jul 17 04:51:22.159017 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.158997    2582 scope.go:117] "RemoveContainer" containerID="9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d"
Jul 17 04:51:22.159319 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:51:22.159300    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d\": container with ID starting with 9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d not found: ID does not exist" containerID="9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d"
Jul 17 04:51:22.159389 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.159332    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d"} err="failed to get container status \"9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d\": rpc error: code = NotFound desc = could not find container \"9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d\": container with ID starting with 9ecb5b92ccc8cdd089a65b1e8d972d55da13fa55fa82eaa98eccf26d0c1d5d2d not found: ID does not exist"
Jul 17 04:51:22.170224 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.170174    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ffnhs" podStartSLOduration=3.9661591830000003 podStartE2EDuration="5.170155423s" totalImagesPullingTime="1.131421321s" totalInitContainerRuntime="72.574919ms" isStatefulPod=true podCreationTimestamp="2026-07-17 04:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:51:22.169742685 +0000 UTC m=+85.906805022" watchObservedRunningTime="2026-07-17 04:51:22.170155423 +0000 UTC m=+85.907217760"
Jul 17 04:51:22.184506 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.184478    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6cd5d478-8cfdf"]
Jul 17 04:51:22.188639 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.188613    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f6cd5d478-8cfdf"]
Jul 17 04:51:22.728989 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:22.728951    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" path="/var/lib/kubelet/pods/aed5dbbc-7e65-4ddb-8512-a013594b38eb/volumes"
Jul 17 04:51:42.106705 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:51:42.106672    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wlb4b"
Jul 17 04:54:23.904657 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.904618    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"]
Jul 17 04:54:23.905301 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.904937    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" containerName="registry"
Jul 17 04:54:23.905301 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.904954    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" containerName="registry"
Jul 17 04:54:23.905301 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.905036    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed5dbbc-7e65-4ddb-8512-a013594b38eb" containerName="registry"
Jul 17 04:54:23.907044 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.907024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:23.909564 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.909540    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 04:54:23.909671 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.909590    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gfmxw\""
Jul 17 04:54:23.910633 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.910614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 04:54:23.917021 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:23.916995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"]
Jul 17 04:54:24.020273 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.020229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3c3e2c4-757a-4710-a4ad-9f973a1225f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gpmnf\" (UID: \"e3c3e2c4-757a-4710-a4ad-9f973a1225f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.020467 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.020313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvm4\" (UniqueName: \"kubernetes.io/projected/e3c3e2c4-757a-4710-a4ad-9f973a1225f4-kube-api-access-mxvm4\") pod \"openshift-lws-operator-fd8ccff4c-gpmnf\" (UID: \"e3c3e2c4-757a-4710-a4ad-9f973a1225f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.120877 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.120828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvm4\" (UniqueName: \"kubernetes.io/projected/e3c3e2c4-757a-4710-a4ad-9f973a1225f4-kube-api-access-mxvm4\") pod \"openshift-lws-operator-fd8ccff4c-gpmnf\" (UID: \"e3c3e2c4-757a-4710-a4ad-9f973a1225f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.121032 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.120918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3c3e2c4-757a-4710-a4ad-9f973a1225f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gpmnf\" (UID: \"e3c3e2c4-757a-4710-a4ad-9f973a1225f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.121227 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.121213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3c3e2c4-757a-4710-a4ad-9f973a1225f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gpmnf\" (UID: \"e3c3e2c4-757a-4710-a4ad-9f973a1225f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.129516 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.129495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvm4\" (UniqueName: \"kubernetes.io/projected/e3c3e2c4-757a-4710-a4ad-9f973a1225f4-kube-api-access-mxvm4\") pod \"openshift-lws-operator-fd8ccff4c-gpmnf\" (UID: \"e3c3e2c4-757a-4710-a4ad-9f973a1225f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.216526 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.216482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"
Jul 17 04:54:24.344697 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.344575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf"]
Jul 17 04:54:24.347400 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:54:24.347373    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3c3e2c4_757a_4710_a4ad_9f973a1225f4.slice/crio-1484e201fb8c184297358de86258f9236fadbc4f51f2db6d096d55d5bbfbd5f1 WatchSource:0}: Error finding container 1484e201fb8c184297358de86258f9236fadbc4f51f2db6d096d55d5bbfbd5f1: Status 404 returned error can't find the container with id 1484e201fb8c184297358de86258f9236fadbc4f51f2db6d096d55d5bbfbd5f1
Jul 17 04:54:24.763722 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:24.763693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf" event={"ID":"e3c3e2c4-757a-4710-a4ad-9f973a1225f4","Type":"ContainerStarted","Data":"1484e201fb8c184297358de86258f9236fadbc4f51f2db6d096d55d5bbfbd5f1"}
Jul 17 04:54:26.772665 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:26.772630    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf" event={"ID":"e3c3e2c4-757a-4710-a4ad-9f973a1225f4","Type":"ContainerStarted","Data":"58eea8c0dc89ccf53b8e8c94cdad53b382bdbad5eab9ddd7868b6befa2566963"}
Jul 17 04:54:26.789240 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:26.789177    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gpmnf" podStartSLOduration=1.490613441 podStartE2EDuration="3.789157251s" totalImagesPullingTime="2.29854381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:54:26.788027888 +0000 UTC m=+270.525090226" watchObservedRunningTime="2026-07-17 04:54:26.789157251 +0000 UTC m=+270.526219586"
Jul 17 04:54:47.018701 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.018664    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"]
Jul 17 04:54:47.021786 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.021764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.026777 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.026749    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 04:54:47.027049 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.027025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mx6ln\""
Jul 17 04:54:47.027177 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.027082    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 04:54:47.027582 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.027565    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 04:54:47.027692 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.027579    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 04:54:47.042159 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.042129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"]
Jul 17 04:54:47.089826 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.089787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea8d7925-3cf2-4715-a627-62285a888f43-tmp\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.089826 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.089828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea8d7925-3cf2-4715-a627-62285a888f43-webhook-cert\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.090038 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.089862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea8d7925-3cf2-4715-a627-62285a888f43-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.090038 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.089893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8x4c\" (UniqueName: \"kubernetes.io/projected/ea8d7925-3cf2-4715-a627-62285a888f43-kube-api-access-d8x4c\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.090038 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.089997    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea8d7925-3cf2-4715-a627-62285a888f43-manifests\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.190416 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.190381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea8d7925-3cf2-4715-a627-62285a888f43-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.190551 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.190421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8x4c\" (UniqueName: \"kubernetes.io/projected/ea8d7925-3cf2-4715-a627-62285a888f43-kube-api-access-d8x4c\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.190551 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.190459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea8d7925-3cf2-4715-a627-62285a888f43-manifests\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.190551 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.190479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea8d7925-3cf2-4715-a627-62285a888f43-tmp\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.190551 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.190500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea8d7925-3cf2-4715-a627-62285a888f43-webhook-cert\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.190855 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.190835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea8d7925-3cf2-4715-a627-62285a888f43-manifests\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.192968 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.192937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea8d7925-3cf2-4715-a627-62285a888f43-tmp\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.193157 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.193140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea8d7925-3cf2-4715-a627-62285a888f43-webhook-cert\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.193157 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.193152    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea8d7925-3cf2-4715-a627-62285a888f43-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.198479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.198456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8x4c\" (UniqueName: \"kubernetes.io/projected/ea8d7925-3cf2-4715-a627-62285a888f43-kube-api-access-d8x4c\") pod \"opendatahub-operator-controller-manager-d857c78d5-qzbdb\" (UID: \"ea8d7925-3cf2-4715-a627-62285a888f43\") " pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.331479 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.331385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:47.470365 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.470293    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"]
Jul 17 04:54:47.473775 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:54:47.473746    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8d7925_3cf2_4715_a627_62285a888f43.slice/crio-be792806d1ba20ba277ea45272baa240d6345413707ed29f4e1c2808f82434ae WatchSource:0}: Error finding container be792806d1ba20ba277ea45272baa240d6345413707ed29f4e1c2808f82434ae: Status 404 returned error can't find the container with id be792806d1ba20ba277ea45272baa240d6345413707ed29f4e1c2808f82434ae
Jul 17 04:54:47.841196 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:47.841157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb" event={"ID":"ea8d7925-3cf2-4715-a627-62285a888f43","Type":"ContainerStarted","Data":"be792806d1ba20ba277ea45272baa240d6345413707ed29f4e1c2808f82434ae"}
Jul 17 04:54:50.856128 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:50.856029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb" event={"ID":"ea8d7925-3cf2-4715-a627-62285a888f43","Type":"ContainerStarted","Data":"98e5a8b7c22bf214ffd000e96316329ad79217ac418d37ecbf311bb47bf5bfb7"}
Jul 17 04:54:51.861385 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:51.861353    2582 generic.go:356] "Generic (PLEG): container finished" podID="ea8d7925-3cf2-4715-a627-62285a888f43" containerID="98e5a8b7c22bf214ffd000e96316329ad79217ac418d37ecbf311bb47bf5bfb7" exitCode=0
Jul 17 04:54:51.861765 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:51.861439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb" event={"ID":"ea8d7925-3cf2-4715-a627-62285a888f43","Type":"ContainerDied","Data":"98e5a8b7c22bf214ffd000e96316329ad79217ac418d37ecbf311bb47bf5bfb7"}
Jul 17 04:54:52.866374 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:52.866333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb" event={"ID":"ea8d7925-3cf2-4715-a627-62285a888f43","Type":"ContainerStarted","Data":"a0231173f06007346d6ab745f078b8c79f32ee219b6494246f08383895395df6"}
Jul 17 04:54:52.866779 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:52.866449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:54:52.889834 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:52.889527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb" podStartSLOduration=2.403793325 podStartE2EDuration="6.889506054s" totalImagesPullingTime="3.400962346s" totalInitContainerRuntime="1.084750383s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:54:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:54:52.887599858 +0000 UTC m=+296.624662217" watchObservedRunningTime="2026-07-17 04:54:52.889506054 +0000 UTC m=+296.626568394"
Jul 17 04:54:56.635332 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:56.635302    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:54:56.638047 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:56.638018    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:54:56.642068 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:54:56.642045    2582 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 04:55:03.873069 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:03.873040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d857c78d5-qzbdb"
Jul 17 04:55:09.632709 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.632677    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"]
Jul 17 04:55:09.637183 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.637160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.639726 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.639697    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 04:55:09.639995 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.639978    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 04:55:09.641119 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.641099    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 04:55:09.641119 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.641115    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xp5n2\""
Jul 17 04:55:09.641295 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.641116    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 04:55:09.646976 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.646950    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"]
Jul 17 04:55:09.761496 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.761458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1d59511-6785-415f-b095-6dc3dd7031d5-tmp\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.761681 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.761521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc22w\" (UniqueName: \"kubernetes.io/projected/e1d59511-6785-415f-b095-6dc3dd7031d5-kube-api-access-jc22w\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.761681 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.761585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d59511-6785-415f-b095-6dc3dd7031d5-tls-certs\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.862524 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.862476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc22w\" (UniqueName: \"kubernetes.io/projected/e1d59511-6785-415f-b095-6dc3dd7031d5-kube-api-access-jc22w\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.862745 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.862553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d59511-6785-415f-b095-6dc3dd7031d5-tls-certs\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.862745 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.862608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1d59511-6785-415f-b095-6dc3dd7031d5-tmp\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.865079 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.865049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1d59511-6785-415f-b095-6dc3dd7031d5-tmp\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.865312 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.865295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d59511-6785-415f-b095-6dc3dd7031d5-tls-certs\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.871106 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.871081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc22w\" (UniqueName: \"kubernetes.io/projected/e1d59511-6785-415f-b095-6dc3dd7031d5-kube-api-access-jc22w\") pod \"kube-auth-proxy-d686d4c64-tcwx9\" (UID: \"e1d59511-6785-415f-b095-6dc3dd7031d5\") " pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:09.947017 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:09.946988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"
Jul 17 04:55:10.089766 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:10.089740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9"]
Jul 17 04:55:10.092325 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:55:10.092298    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d59511_6785_415f_b095_6dc3dd7031d5.slice/crio-2798bf2f068ac621a0f3b1024daed750bb2986b187a935dbeeec1ab2d654cabd WatchSource:0}: Error finding container 2798bf2f068ac621a0f3b1024daed750bb2986b187a935dbeeec1ab2d654cabd: Status 404 returned error can't find the container with id 2798bf2f068ac621a0f3b1024daed750bb2986b187a935dbeeec1ab2d654cabd
Jul 17 04:55:10.094492 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:10.094472    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 04:55:10.934076 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:10.934033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9" event={"ID":"e1d59511-6785-415f-b095-6dc3dd7031d5","Type":"ContainerStarted","Data":"2798bf2f068ac621a0f3b1024daed750bb2986b187a935dbeeec1ab2d654cabd"}
Jul 17 04:55:13.947731 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:13.947692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9" event={"ID":"e1d59511-6785-415f-b095-6dc3dd7031d5","Type":"ContainerStarted","Data":"c9fe3ec50a675cff533ff217cfeeef99831c092ec27884374c0f2e3ae9801445"}
Jul 17 04:55:13.966830 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:13.966771    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d686d4c64-tcwx9" podStartSLOduration=1.874042707 podStartE2EDuration="4.966752303s" totalImagesPullingTime="3.092709596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:55:13.96539506 +0000 UTC m=+317.702457428" watchObservedRunningTime="2026-07-17 04:55:13.966752303 +0000 UTC m=+317.703814639"
Jul 17 04:55:14.690144 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.690111    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"]
Jul 17 04:55:14.693317 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.693300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.697421 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.697398    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-k8t6r\""
Jul 17 04:55:14.697421 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.697416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 04:55:14.697575 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.697434    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 04:55:14.697575 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.697398    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 04:55:14.701840 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.701810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6025461-9594-48ae-a90d-b7a2779befe8-cert\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.701955 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.701851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6025461-9594-48ae-a90d-b7a2779befe8-manager-config\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.701955 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.701886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6025461-9594-48ae-a90d-b7a2779befe8-metrics-cert\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.701955 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.701912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sw5j\" (UniqueName: \"kubernetes.io/projected/b6025461-9594-48ae-a90d-b7a2779befe8-kube-api-access-7sw5j\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.702389 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.702372    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"]
Jul 17 04:55:14.802858 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.802823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6025461-9594-48ae-a90d-b7a2779befe8-metrics-cert\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.802858 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.802866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sw5j\" (UniqueName: \"kubernetes.io/projected/b6025461-9594-48ae-a90d-b7a2779befe8-kube-api-access-7sw5j\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.803114 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.802920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6025461-9594-48ae-a90d-b7a2779befe8-cert\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.803114 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.802941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6025461-9594-48ae-a90d-b7a2779befe8-manager-config\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.803738 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.803715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6025461-9594-48ae-a90d-b7a2779befe8-manager-config\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.806407 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.806384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6025461-9594-48ae-a90d-b7a2779befe8-metrics-cert\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.806639 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.806607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6025461-9594-48ae-a90d-b7a2779befe8-cert\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:14.813720 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:14.813690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sw5j\" (UniqueName: \"kubernetes.io/projected/b6025461-9594-48ae-a90d-b7a2779befe8-kube-api-access-7sw5j\") pod \"lws-controller-manager-559bdd8b4c-6crsw\" (UID: \"b6025461-9594-48ae-a90d-b7a2779befe8\") " pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:15.002646 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:15.002536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:15.136317 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:15.136290    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"]
Jul 17 04:55:15.139666 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:55:15.139635    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6025461_9594_48ae_a90d_b7a2779befe8.slice/crio-32149f605fca6840dc1950af1be811e4a9a182dada662948b53d1f9480808766 WatchSource:0}: Error finding container 32149f605fca6840dc1950af1be811e4a9a182dada662948b53d1f9480808766: Status 404 returned error can't find the container with id 32149f605fca6840dc1950af1be811e4a9a182dada662948b53d1f9480808766
Jul 17 04:55:15.959174 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:15.959138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw" event={"ID":"b6025461-9594-48ae-a90d-b7a2779befe8","Type":"ContainerStarted","Data":"32149f605fca6840dc1950af1be811e4a9a182dada662948b53d1f9480808766"}
Jul 17 04:55:16.969178 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:16.969142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw" event={"ID":"b6025461-9594-48ae-a90d-b7a2779befe8","Type":"ContainerStarted","Data":"95dc7d749d81429e30e4373b3f36fb9d9178a5f6b67ff2105e129e78c3136bb1"}
Jul 17 04:55:16.969558 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:16.969380    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:55:17.002373 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:17.002326    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw" podStartSLOduration=1.512670803 podStartE2EDuration="3.002308992s" totalImagesPullingTime="1.489638189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:55:17.00159876 +0000 UTC m=+320.738661095" watchObservedRunningTime="2026-07-17 04:55:17.002308992 +0000 UTC m=+320.739371402"
Jul 17 04:55:27.976157 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:55:27.976118    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-559bdd8b4c-6crsw"
Jul 17 04:58:00.434085 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.434048    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rh84q"]
Jul 17 04:58:00.437004 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.436989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:00.439499 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.439473    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ljw4m\""
Jul 17 04:58:00.440573 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.440547    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 04:58:00.440680 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.440617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 04:58:00.444940 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.444915    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rh84q"]
Jul 17 04:58:00.559122 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.559087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdj6k\" (UniqueName: \"kubernetes.io/projected/560de1d7-90b6-474a-9567-4276c243370d-kube-api-access-pdj6k\") pod \"authorino-58cd955675-rh84q\" (UID: \"560de1d7-90b6-474a-9567-4276c243370d\") " pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:00.593684 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.593649    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-q6x5r"]
Jul 17 04:58:00.596771 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.596751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:00.603888 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.603868    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q6x5r"]
Jul 17 04:58:00.660202 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.660161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdj6k\" (UniqueName: \"kubernetes.io/projected/560de1d7-90b6-474a-9567-4276c243370d-kube-api-access-pdj6k\") pod \"authorino-58cd955675-rh84q\" (UID: \"560de1d7-90b6-474a-9567-4276c243370d\") " pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:00.668550 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.668523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdj6k\" (UniqueName: \"kubernetes.io/projected/560de1d7-90b6-474a-9567-4276c243370d-kube-api-access-pdj6k\") pod \"authorino-58cd955675-rh84q\" (UID: \"560de1d7-90b6-474a-9567-4276c243370d\") " pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:00.746848 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.746817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:00.761032 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.761005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp6rq\" (UniqueName: \"kubernetes.io/projected/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba-kube-api-access-cp6rq\") pod \"authorino-75d556bd77-q6x5r\" (UID: \"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba\") " pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:00.861677 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.861644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp6rq\" (UniqueName: \"kubernetes.io/projected/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba-kube-api-access-cp6rq\") pod \"authorino-75d556bd77-q6x5r\" (UID: \"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba\") " pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:00.873450 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.873423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp6rq\" (UniqueName: \"kubernetes.io/projected/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba-kube-api-access-cp6rq\") pod \"authorino-75d556bd77-q6x5r\" (UID: \"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba\") " pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:00.875952 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.875852    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rh84q"]
Jul 17 04:58:00.878416 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:58:00.878387    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod560de1d7_90b6_474a_9567_4276c243370d.slice/crio-51390b01e243f252315ea30a6651f1b653a5c14c2d6ebdb8fc8c42241b3cf9e9 WatchSource:0}: Error finding container 51390b01e243f252315ea30a6651f1b653a5c14c2d6ebdb8fc8c42241b3cf9e9: Status 404 returned error can't find the container with id 51390b01e243f252315ea30a6651f1b653a5c14c2d6ebdb8fc8c42241b3cf9e9
Jul 17 04:58:00.905726 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:00.905693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:01.036474 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:01.036226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q6x5r"]
Jul 17 04:58:01.038868 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:58:01.038842    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod454d6a6a_fbee_46cb_ac42_7d5f86bdd7ba.slice/crio-a2f67e73b30a1428db8ca25ec53f4007fbae1b6775da8b8a1e9666e821b0364c WatchSource:0}: Error finding container a2f67e73b30a1428db8ca25ec53f4007fbae1b6775da8b8a1e9666e821b0364c: Status 404 returned error can't find the container with id a2f67e73b30a1428db8ca25ec53f4007fbae1b6775da8b8a1e9666e821b0364c
Jul 17 04:58:01.547684 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:01.547641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q6x5r" event={"ID":"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba","Type":"ContainerStarted","Data":"a2f67e73b30a1428db8ca25ec53f4007fbae1b6775da8b8a1e9666e821b0364c"}
Jul 17 04:58:01.549301 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:01.549211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rh84q" event={"ID":"560de1d7-90b6-474a-9567-4276c243370d","Type":"ContainerStarted","Data":"51390b01e243f252315ea30a6651f1b653a5c14c2d6ebdb8fc8c42241b3cf9e9"}
Jul 17 04:58:04.569412 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:04.569369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rh84q" event={"ID":"560de1d7-90b6-474a-9567-4276c243370d","Type":"ContainerStarted","Data":"7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c"}
Jul 17 04:58:04.571403 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:04.571377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q6x5r" event={"ID":"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba","Type":"ContainerStarted","Data":"52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460"}
Jul 17 04:58:04.586795 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:04.586734    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rh84q" podStartSLOduration=1.011878025 podStartE2EDuration="4.586688672s" totalImagesPullingTime="3.574810647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:58:04.584880687 +0000 UTC m=+488.321943023" watchObservedRunningTime="2026-07-17 04:58:04.586688672 +0000 UTC m=+488.323751009"
Jul 17 04:58:04.603437 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:04.603339    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-q6x5r" podStartSLOduration=1.176044291 podStartE2EDuration="4.603320037s" totalImagesPullingTime="3.427275746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:58:04.601874656 +0000 UTC m=+488.338936990" watchObservedRunningTime="2026-07-17 04:58:04.603320037 +0000 UTC m=+488.340382411"
Jul 17 04:58:04.626446 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:04.626415    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rh84q"]
Jul 17 04:58:06.579242 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:06.579207    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rh84q" podUID="560de1d7-90b6-474a-9567-4276c243370d" containerName="authorino" containerID="cri-o://7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c" gracePeriod=30
Jul 17 04:58:06.827494 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:06.827468    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:06.904881 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:06.904793    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdj6k\" (UniqueName: \"kubernetes.io/projected/560de1d7-90b6-474a-9567-4276c243370d-kube-api-access-pdj6k\") pod \"560de1d7-90b6-474a-9567-4276c243370d\" (UID: \"560de1d7-90b6-474a-9567-4276c243370d\") "
Jul 17 04:58:06.907073 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:06.907046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560de1d7-90b6-474a-9567-4276c243370d-kube-api-access-pdj6k" (OuterVolumeSpecName: "kube-api-access-pdj6k") pod "560de1d7-90b6-474a-9567-4276c243370d" (UID: "560de1d7-90b6-474a-9567-4276c243370d"). InnerVolumeSpecName "kube-api-access-pdj6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 04:58:07.006099 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.006066    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdj6k\" (UniqueName: \"kubernetes.io/projected/560de1d7-90b6-474a-9567-4276c243370d-kube-api-access-pdj6k\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:58:07.584164 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.584129    2582 generic.go:356] "Generic (PLEG): container finished" podID="560de1d7-90b6-474a-9567-4276c243370d" containerID="7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c" exitCode=0
Jul 17 04:58:07.584164 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.584167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rh84q"
Jul 17 04:58:07.584707 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.584205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rh84q" event={"ID":"560de1d7-90b6-474a-9567-4276c243370d","Type":"ContainerDied","Data":"7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c"}
Jul 17 04:58:07.584707 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.584236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rh84q" event={"ID":"560de1d7-90b6-474a-9567-4276c243370d","Type":"ContainerDied","Data":"51390b01e243f252315ea30a6651f1b653a5c14c2d6ebdb8fc8c42241b3cf9e9"}
Jul 17 04:58:07.584707 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.584254    2582 scope.go:117] "RemoveContainer" containerID="7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c"
Jul 17 04:58:07.592346 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.592329    2582 scope.go:117] "RemoveContainer" containerID="7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c"
Jul 17 04:58:07.592600 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:58:07.592580    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c\": container with ID starting with 7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c not found: ID does not exist" containerID="7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c"
Jul 17 04:58:07.592673 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.592608    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c"} err="failed to get container status \"7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c\": rpc error: code = NotFound desc = could not find container \"7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c\": container with ID starting with 7ac2b38f35c5498247849248ffe9d290056d6e5bd852f0b04546dd2747f3aa7c not found: ID does not exist"
Jul 17 04:58:07.603745 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.603722    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rh84q"]
Jul 17 04:58:07.607635 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:07.607609    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rh84q"]
Jul 17 04:58:08.730142 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:08.730098    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560de1d7-90b6-474a-9567-4276c243370d" path="/var/lib/kubelet/pods/560de1d7-90b6-474a-9567-4276c243370d/volumes"
Jul 17 04:58:33.789811 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.789780    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xwkkm"]
Jul 17 04:58:33.790171 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.790047    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560de1d7-90b6-474a-9567-4276c243370d" containerName="authorino"
Jul 17 04:58:33.790171 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.790057    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="560de1d7-90b6-474a-9567-4276c243370d" containerName="authorino"
Jul 17 04:58:33.790171 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.790110    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="560de1d7-90b6-474a-9567-4276c243370d" containerName="authorino"
Jul 17 04:58:33.792828 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.792811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:33.800056 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.799884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xwkkm"]
Jul 17 04:58:33.907801 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:33.907768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjbw\" (UniqueName: \"kubernetes.io/projected/13a79ec0-ca95-4d0f-9c6d-705073888364-kube-api-access-4rjbw\") pod \"authorino-5d4b67c446-xwkkm\" (UID: \"13a79ec0-ca95-4d0f-9c6d-705073888364\") " pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:34.001607 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.001574    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xwkkm"]
Jul 17 04:58:34.001826 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:58:34.001807    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4rjbw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-xwkkm" podUID="13a79ec0-ca95-4d0f-9c6d-705073888364"
Jul 17 04:58:34.008352 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.008327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjbw\" (UniqueName: \"kubernetes.io/projected/13a79ec0-ca95-4d0f-9c6d-705073888364-kube-api-access-4rjbw\") pod \"authorino-5d4b67c446-xwkkm\" (UID: \"13a79ec0-ca95-4d0f-9c6d-705073888364\") " pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:34.016227 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.016202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjbw\" (UniqueName: \"kubernetes.io/projected/13a79ec0-ca95-4d0f-9c6d-705073888364-kube-api-access-4rjbw\") pod \"authorino-5d4b67c446-xwkkm\" (UID: \"13a79ec0-ca95-4d0f-9c6d-705073888364\") " pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:34.030039 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.030005    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dcb66dbcc-5vbnk"]
Jul 17 04:58:34.032994 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.032975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.035448 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.035424    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 04:58:34.040282 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.040199    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dcb66dbcc-5vbnk"]
Jul 17 04:58:34.109019 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.108981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2wj\" (UniqueName: \"kubernetes.io/projected/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-kube-api-access-zr2wj\") pod \"authorino-7dcb66dbcc-5vbnk\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") " pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.109195 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.109034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-tls-cert\") pod \"authorino-7dcb66dbcc-5vbnk\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") " pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.209488 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.209451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2wj\" (UniqueName: \"kubernetes.io/projected/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-kube-api-access-zr2wj\") pod \"authorino-7dcb66dbcc-5vbnk\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") " pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.209652 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.209500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-tls-cert\") pod \"authorino-7dcb66dbcc-5vbnk\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") " pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.212031 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.212012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-tls-cert\") pod \"authorino-7dcb66dbcc-5vbnk\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") " pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.217338 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.217312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2wj\" (UniqueName: \"kubernetes.io/projected/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-kube-api-access-zr2wj\") pod \"authorino-7dcb66dbcc-5vbnk\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") " pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.342471 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.342371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 04:58:34.476199 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.476173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dcb66dbcc-5vbnk"]
Jul 17 04:58:34.478710 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:58:34.478677    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25caeaa6_00b8_4c9b_9729_f2d5bd42ab60.slice/crio-ce97293771130ac26f7530435f120ecccf907d8668930e81cf82a9647cc37e97 WatchSource:0}: Error finding container ce97293771130ac26f7530435f120ecccf907d8668930e81cf82a9647cc37e97: Status 404 returned error can't find the container with id ce97293771130ac26f7530435f120ecccf907d8668930e81cf82a9647cc37e97
Jul 17 04:58:34.690708 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.690684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:34.690871 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.690688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk" event={"ID":"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60","Type":"ContainerStarted","Data":"ce97293771130ac26f7530435f120ecccf907d8668930e81cf82a9647cc37e97"}
Jul 17 04:58:34.698792 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.698767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:34.814513 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.814480    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rjbw\" (UniqueName: \"kubernetes.io/projected/13a79ec0-ca95-4d0f-9c6d-705073888364-kube-api-access-4rjbw\") pod \"13a79ec0-ca95-4d0f-9c6d-705073888364\" (UID: \"13a79ec0-ca95-4d0f-9c6d-705073888364\") "
Jul 17 04:58:34.816804 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.816785    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a79ec0-ca95-4d0f-9c6d-705073888364-kube-api-access-4rjbw" (OuterVolumeSpecName: "kube-api-access-4rjbw") pod "13a79ec0-ca95-4d0f-9c6d-705073888364" (UID: "13a79ec0-ca95-4d0f-9c6d-705073888364"). InnerVolumeSpecName "kube-api-access-4rjbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 04:58:34.915751 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:34.915720    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rjbw\" (UniqueName: \"kubernetes.io/projected/13a79ec0-ca95-4d0f-9c6d-705073888364-kube-api-access-4rjbw\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:58:35.700004 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.699971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk" event={"ID":"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60","Type":"ContainerStarted","Data":"54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41"}
Jul 17 04:58:35.700195 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.700093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xwkkm"
Jul 17 04:58:35.718577 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.718527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk" podStartSLOduration=0.924100928 podStartE2EDuration="1.718512231s" totalImagesPullingTime="794.411303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:58:35.717308694 +0000 UTC m=+519.454371030" watchObservedRunningTime="2026-07-17 04:58:35.718512231 +0000 UTC m=+519.455574567"
Jul 17 04:58:35.742504 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.742471    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xwkkm"]
Jul 17 04:58:35.749062 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.749034    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q6x5r"]
Jul 17 04:58:35.749327 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.749291    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-q6x5r" podUID="454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" containerName="authorino" containerID="cri-o://52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460" gracePeriod=30
Jul 17 04:58:35.751657 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.751631    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xwkkm"]
Jul 17 04:58:35.973805 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:35.973781    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:36.026949 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.026925    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp6rq\" (UniqueName: \"kubernetes.io/projected/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba-kube-api-access-cp6rq\") pod \"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba\" (UID: \"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba\") "
Jul 17 04:58:36.029098 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.029076    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba-kube-api-access-cp6rq" (OuterVolumeSpecName: "kube-api-access-cp6rq") pod "454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" (UID: "454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba"). InnerVolumeSpecName "kube-api-access-cp6rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 04:58:36.128340 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.128304    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp6rq\" (UniqueName: \"kubernetes.io/projected/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba-kube-api-access-cp6rq\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 04:58:36.705040 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.705001    2582 generic.go:356] "Generic (PLEG): container finished" podID="454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" containerID="52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460" exitCode=0
Jul 17 04:58:36.705237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.705046    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q6x5r"
Jul 17 04:58:36.705237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.705098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q6x5r" event={"ID":"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba","Type":"ContainerDied","Data":"52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460"}
Jul 17 04:58:36.705237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.705141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q6x5r" event={"ID":"454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba","Type":"ContainerDied","Data":"a2f67e73b30a1428db8ca25ec53f4007fbae1b6775da8b8a1e9666e821b0364c"}
Jul 17 04:58:36.705237 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.705164    2582 scope.go:117] "RemoveContainer" containerID="52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460"
Jul 17 04:58:36.713776 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.713758    2582 scope.go:117] "RemoveContainer" containerID="52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460"
Jul 17 04:58:36.714050 ip-10-0-140-139 kubenswrapper[2582]: E0717 04:58:36.714033    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460\": container with ID starting with 52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460 not found: ID does not exist" containerID="52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460"
Jul 17 04:58:36.714092 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.714059    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460"} err="failed to get container status \"52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460\": rpc error: code = NotFound desc = could not find container \"52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460\": container with ID starting with 52beb87083d3c196367df1ca14be41e9efd183eb40ca788a5f5e0aeedbc5e460 not found: ID does not exist"
Jul 17 04:58:36.724603 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.724571    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q6x5r"]
Jul 17 04:58:36.730593 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.730561    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a79ec0-ca95-4d0f-9c6d-705073888364" path="/var/lib/kubelet/pods/13a79ec0-ca95-4d0f-9c6d-705073888364/volumes"
Jul 17 04:58:36.730804 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:36.730789    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q6x5r"]
Jul 17 04:58:38.728952 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:58:38.728915    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" path="/var/lib/kubelet/pods/454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba/volumes"
Jul 17 04:59:17.941817 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.941777    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-fnxlv"]
Jul 17 04:59:17.942318 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.942145    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" containerName="authorino"
Jul 17 04:59:17.942318 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.942161    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" containerName="authorino"
Jul 17 04:59:17.942318 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.942252    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="454d6a6a-fbee-46cb-ac42-7d5f86bdd7ba" containerName="authorino"
Jul 17 04:59:17.948818 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.948787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:17.951484 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.951459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ffp7c\""
Jul 17 04:59:17.951614 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.951459    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 17 04:59:17.952698 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.952668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b645dd0-44aa-4aaa-94ff-ac84a6a361d0-plugins-config-volume\") pod \"payload-processing-5bf47cd988-fnxlv\" (UID: \"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0\") " pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:17.952926 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.952903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8db7\" (UniqueName: \"kubernetes.io/projected/0b645dd0-44aa-4aaa-94ff-ac84a6a361d0-kube-api-access-r8db7\") pod \"payload-processing-5bf47cd988-fnxlv\" (UID: \"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0\") " pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:17.954034 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:17.954013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-fnxlv"]
Jul 17 04:59:18.053274 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.053229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b645dd0-44aa-4aaa-94ff-ac84a6a361d0-plugins-config-volume\") pod \"payload-processing-5bf47cd988-fnxlv\" (UID: \"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0\") " pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:18.053451 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.053305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8db7\" (UniqueName: \"kubernetes.io/projected/0b645dd0-44aa-4aaa-94ff-ac84a6a361d0-kube-api-access-r8db7\") pod \"payload-processing-5bf47cd988-fnxlv\" (UID: \"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0\") " pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:18.053987 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.053969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b645dd0-44aa-4aaa-94ff-ac84a6a361d0-plugins-config-volume\") pod \"payload-processing-5bf47cd988-fnxlv\" (UID: \"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0\") " pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:18.062117 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.062094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8db7\" (UniqueName: \"kubernetes.io/projected/0b645dd0-44aa-4aaa-94ff-ac84a6a361d0-kube-api-access-r8db7\") pod \"payload-processing-5bf47cd988-fnxlv\" (UID: \"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0\") " pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:18.259630 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.259531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:18.398491 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.398464    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-fnxlv"]
Jul 17 04:59:18.400998 ip-10-0-140-139 kubenswrapper[2582]: W0717 04:59:18.400972    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b645dd0_44aa_4aaa_94ff_ac84a6a361d0.slice/crio-ec71122a93d638b5abc09494db21fe495ed4052573cdd6c76b0efd79a1a7d769 WatchSource:0}: Error finding container ec71122a93d638b5abc09494db21fe495ed4052573cdd6c76b0efd79a1a7d769: Status 404 returned error can't find the container with id ec71122a93d638b5abc09494db21fe495ed4052573cdd6c76b0efd79a1a7d769
Jul 17 04:59:18.863446 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:18.863409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv" event={"ID":"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0","Type":"ContainerStarted","Data":"ec71122a93d638b5abc09494db21fe495ed4052573cdd6c76b0efd79a1a7d769"}
Jul 17 04:59:20.874187 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:20.874102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv" event={"ID":"0b645dd0-44aa-4aaa-94ff-ac84a6a361d0","Type":"ContainerStarted","Data":"9c563425168ceb10f27ea60d5d09752dc72ed2ec24e06f5564d1be6afffbd565"}
Jul 17 04:59:20.874582 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:20.874241    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:20.891081 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:20.891023    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv" podStartSLOduration=1.703971985 podStartE2EDuration="3.891006997s" totalImagesPullingTime="2.187035012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 04:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 04:59:20.889185265 +0000 UTC m=+564.626247601" watchObservedRunningTime="2026-07-17 04:59:20.891006997 +0000 UTC m=+564.628069333"
Jul 17 04:59:31.881053 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:31.881022    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-fnxlv"
Jul 17 04:59:56.660019 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:56.659989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 04:59:56.660576 ip-10-0-140-139 kubenswrapper[2582]: I0717 04:59:56.660307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:00:00.142522 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:00.142471    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"]
Jul 17 05:00:00.150178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:00.150155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"
Jul 17 05:00:00.152834 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:00.152809    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:00:00.156121 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:00.156096    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"]
Jul 17 05:00:00.162552 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:00.162530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"
Jul 17 05:00:00.302204 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:00.302176    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"]
Jul 17 05:00:00.304892 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:00:00.304866    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8848042c_f164_4766_8ed3_006ae73fa1bc.slice/crio-e20a19ed2515b63dfbd57ee656ffb451456e3909ec0b3e851bf59e01b56c234f WatchSource:0}: Error finding container e20a19ed2515b63dfbd57ee656ffb451456e3909ec0b3e851bf59e01b56c234f: Status 404 returned error can't find the container with id e20a19ed2515b63dfbd57ee656ffb451456e3909ec0b3e851bf59e01b56c234f
Jul 17 05:00:01.040274 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:01.040227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf" event={"ID":"8848042c-f164-4766-8ed3-006ae73fa1bc","Type":"ContainerStarted","Data":"e20a19ed2515b63dfbd57ee656ffb451456e3909ec0b3e851bf59e01b56c234f"}
Jul 17 05:00:18.109492 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:18.109457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf" event={"ID":"8848042c-f164-4766-8ed3-006ae73fa1bc","Type":"ContainerStarted","Data":"6c8b270ddcc219bdc42701ee8c8742670131f24c31570c4fe76b1178ed7ec1b3"}
Jul 17 05:00:18.122776 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:18.122714    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf" podStartSLOduration=0.842242083 podStartE2EDuration="18.122693629s" totalImagesPullingTime="17.280451546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:00:18.122417114 +0000 UTC m=+621.859479482" watchObservedRunningTime="2026-07-17 05:00:18.122693629 +0000 UTC m=+621.859755966"
Jul 17 05:00:19.115416 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:19.115381    2582 generic.go:356] "Generic (PLEG): container finished" podID="8848042c-f164-4766-8ed3-006ae73fa1bc" containerID="6c8b270ddcc219bdc42701ee8c8742670131f24c31570c4fe76b1178ed7ec1b3" exitCode=0
Jul 17 05:00:19.115806 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:19.115476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf" event={"ID":"8848042c-f164-4766-8ed3-006ae73fa1bc","Type":"ContainerDied","Data":"6c8b270ddcc219bdc42701ee8c8742670131f24c31570c4fe76b1178ed7ec1b3"}
Jul 17 05:00:20.248567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:20.248540    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"
Jul 17 05:00:21.124815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:21.124731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf" event={"ID":"8848042c-f164-4766-8ed3-006ae73fa1bc","Type":"ContainerDied","Data":"e20a19ed2515b63dfbd57ee656ffb451456e3909ec0b3e851bf59e01b56c234f"}
Jul 17 05:00:21.124815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:21.124768    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"
Jul 17 05:00:21.124815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:21.124779    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20a19ed2515b63dfbd57ee656ffb451456e3909ec0b3e851bf59e01b56c234f"
Jul 17 05:00:42.836327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.836292    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"]
Jul 17 05:00:42.836874 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.836552    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8848042c-f164-4766-8ed3-006ae73fa1bc" containerName="cleanup"
Jul 17 05:00:42.836874 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.836563    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8848042c-f164-4766-8ed3-006ae73fa1bc" containerName="cleanup"
Jul 17 05:00:42.836874 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.836630    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8848042c-f164-4766-8ed3-006ae73fa1bc" containerName="cleanup"
Jul 17 05:00:42.839628 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.839611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:42.842488 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.842465    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 05:00:42.843449 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.843416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 17 05:00:42.843543 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.843523    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 05:00:42.843891 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.843872    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q7v2x\""
Jul 17 05:00:42.849992 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.849967    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"]
Jul 17 05:00:42.910636 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.910601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:42.910794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.910643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29m4\" (UniqueName: \"kubernetes.io/projected/f4881ea1-ca67-462c-b5d7-23807a56d489-kube-api-access-n29m4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:42.910794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.910736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:42.910794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.910787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:42.910912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.910808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:42.910912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:42.910845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4881ea1-ca67-462c-b5d7-23807a56d489-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.011603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.011567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4881ea1-ca67-462c-b5d7-23807a56d489-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.011795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.011610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.011795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.011634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29m4\" (UniqueName: \"kubernetes.io/projected/f4881ea1-ca67-462c-b5d7-23807a56d489-kube-api-access-n29m4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.011795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.011669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.011795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.011686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.011795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.011703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.012060 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.012036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.012122 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.012096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.012177 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.012162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.014023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.014006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4881ea1-ca67-462c-b5d7-23807a56d489-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.014295 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.014275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4881ea1-ca67-462c-b5d7-23807a56d489-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.020582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.020561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29m4\" (UniqueName: \"kubernetes.io/projected/f4881ea1-ca67-462c-b5d7-23807a56d489-kube-api-access-n29m4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs\" (UID: \"f4881ea1-ca67-462c-b5d7-23807a56d489\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.151892 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.151796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:43.282027 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.281996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"]
Jul 17 05:00:43.284991 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:00:43.284964    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4881ea1_ca67_462c_b5d7_23807a56d489.slice/crio-785322385a12db5e585808d672457786e137ccce44039ebf68cf86c678e03c8b WatchSource:0}: Error finding container 785322385a12db5e585808d672457786e137ccce44039ebf68cf86c678e03c8b: Status 404 returned error can't find the container with id 785322385a12db5e585808d672457786e137ccce44039ebf68cf86c678e03c8b
Jul 17 05:00:43.286316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:43.286300    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 05:00:44.217482 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:44.217416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs" event={"ID":"f4881ea1-ca67-462c-b5d7-23807a56d489","Type":"ContainerStarted","Data":"785322385a12db5e585808d672457786e137ccce44039ebf68cf86c678e03c8b"}
Jul 17 05:00:45.224888 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:45.224845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs" event={"ID":"f4881ea1-ca67-462c-b5d7-23807a56d489","Type":"ContainerStarted","Data":"76ad5c3ddeb32663c22a5a1171180c699e027bcc6596147998303bed2b50244d"}
Jul 17 05:00:45.243728 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:45.243683    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs" podStartSLOduration=2.200081402 podStartE2EDuration="3.24366856s" totalImagesPullingTime="1.043587158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:00:45.241582256 +0000 UTC m=+648.978644609" watchObservedRunningTime="2026-07-17 05:00:45.24366856 +0000 UTC m=+648.980730897"
Jul 17 05:00:53.152910 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:53.152868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:53.153447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:53.152966    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:53.165330 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:53.165307    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:00:53.269028 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:00:53.268997    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs"
Jul 17 05:01:03.027603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.027566    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"]
Jul 17 05:01:03.034730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.034707    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.037326 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.037306    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 17 05:01:03.041197 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.041171    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"]
Jul 17 05:01:03.070642 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.070606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.070802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.070657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d9v\" (UniqueName: \"kubernetes.io/projected/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-kube-api-access-f9d9v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.070802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.070721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.070802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.070766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.070802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.070790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.070934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.070809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.171650 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.171615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.171838 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.171669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d9v\" (UniqueName: \"kubernetes.io/projected/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-kube-api-access-f9d9v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.171838 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.171782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.171838 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.171819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.171995 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.171847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.171995 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.171960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.172099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.172002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.172099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.172071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.172206 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.172190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.174347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.174330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.174558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.174541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.179999 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.179979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d9v\" (UniqueName: \"kubernetes.io/projected/6faecf39-ede8-428b-b1d3-a4ebffc5ecfc-kube-api-access-f9d9v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-td42d\" (UID: \"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.346522 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.346423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:03.482926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:03.482819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"]
Jul 17 05:01:03.485748 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:01:03.485717    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faecf39_ede8_428b_b1d3_a4ebffc5ecfc.slice/crio-9eb6c7cf38f295ce5cd67611fa81e7220b24452240099c1039bcbe16a3daac71 WatchSource:0}: Error finding container 9eb6c7cf38f295ce5cd67611fa81e7220b24452240099c1039bcbe16a3daac71: Status 404 returned error can't find the container with id 9eb6c7cf38f295ce5cd67611fa81e7220b24452240099c1039bcbe16a3daac71
Jul 17 05:01:04.301166 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:04.301133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d" event={"ID":"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc","Type":"ContainerStarted","Data":"a6f1777d9922ce21b8ab9a9212c63a22ee527428fc5cf0bfd29daa348ca0f766"}
Jul 17 05:01:04.301838 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:04.301221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d" event={"ID":"6faecf39-ede8-428b-b1d3-a4ebffc5ecfc","Type":"ContainerStarted","Data":"9eb6c7cf38f295ce5cd67611fa81e7220b24452240099c1039bcbe16a3daac71"}
Jul 17 05:01:04.324118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:04.324067    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d" podStartSLOduration=1.105064475 podStartE2EDuration="1.324052763s" totalImagesPullingTime="218.988288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:01:04.323904958 +0000 UTC m=+668.060967294" watchObservedRunningTime="2026-07-17 05:01:04.324052763 +0000 UTC m=+668.061115167"
Jul 17 05:01:13.347500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:13.347468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:13.348022 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:13.347527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:13.359993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:13.359963    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:01:14.360589 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:01:14.360556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-td42d"
Jul 17 05:02:32.600407 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.600375    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d874fd7c6-rcgtc"]
Jul 17 05:02:32.603709 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.603691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.605444 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.605419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vv48\" (UniqueName: \"kubernetes.io/projected/de593a03-4536-413f-b011-b930e1d849eb-kube-api-access-9vv48\") pod \"authorino-6d874fd7c6-rcgtc\" (UID: \"de593a03-4536-413f-b011-b930e1d849eb\") " pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.605540 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.605469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de593a03-4536-413f-b011-b930e1d849eb-tls-cert\") pod \"authorino-6d874fd7c6-rcgtc\" (UID: \"de593a03-4536-413f-b011-b930e1d849eb\") " pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.613013 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.612993    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d874fd7c6-rcgtc"]
Jul 17 05:02:32.706234 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.706199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vv48\" (UniqueName: \"kubernetes.io/projected/de593a03-4536-413f-b011-b930e1d849eb-kube-api-access-9vv48\") pod \"authorino-6d874fd7c6-rcgtc\" (UID: \"de593a03-4536-413f-b011-b930e1d849eb\") " pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.706234 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.706241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de593a03-4536-413f-b011-b930e1d849eb-tls-cert\") pod \"authorino-6d874fd7c6-rcgtc\" (UID: \"de593a03-4536-413f-b011-b930e1d849eb\") " pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.709078 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.709052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de593a03-4536-413f-b011-b930e1d849eb-tls-cert\") pod \"authorino-6d874fd7c6-rcgtc\" (UID: \"de593a03-4536-413f-b011-b930e1d849eb\") " pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.714558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.714532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vv48\" (UniqueName: \"kubernetes.io/projected/de593a03-4536-413f-b011-b930e1d849eb-kube-api-access-9vv48\") pod \"authorino-6d874fd7c6-rcgtc\" (UID: \"de593a03-4536-413f-b011-b930e1d849eb\") " pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:32.912314 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:32.912200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d874fd7c6-rcgtc"
Jul 17 05:02:33.048783 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.048754    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d874fd7c6-rcgtc"]
Jul 17 05:02:33.051288 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:02:33.051238    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde593a03_4536_413f_b011_b930e1d849eb.slice/crio-8ff77fbbf5b9382b9b967366c4c1567d3ebae2e5500716d35b32438e1d089ec9 WatchSource:0}: Error finding container 8ff77fbbf5b9382b9b967366c4c1567d3ebae2e5500716d35b32438e1d089ec9: Status 404 returned error can't find the container with id 8ff77fbbf5b9382b9b967366c4c1567d3ebae2e5500716d35b32438e1d089ec9
Jul 17 05:02:33.684252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.684220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d874fd7c6-rcgtc" event={"ID":"de593a03-4536-413f-b011-b930e1d849eb","Type":"ContainerStarted","Data":"efd1457a19a125f70e354b209e85ac39f47b543f99532ef62f7c04addb5e5ebd"}
Jul 17 05:02:33.684702 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.684282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d874fd7c6-rcgtc" event={"ID":"de593a03-4536-413f-b011-b930e1d849eb","Type":"ContainerStarted","Data":"8ff77fbbf5b9382b9b967366c4c1567d3ebae2e5500716d35b32438e1d089ec9"}
Jul 17 05:02:33.700630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.700563    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d874fd7c6-rcgtc" podStartSLOduration=1.1984515789999999 podStartE2EDuration="1.700540571s" totalImagesPullingTime="502.088992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:02:33.699604423 +0000 UTC m=+757.436666759" watchObservedRunningTime="2026-07-17 05:02:33.700540571 +0000 UTC m=+757.437602909"
Jul 17 05:02:33.726402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.726372    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dcb66dbcc-5vbnk"]
Jul 17 05:02:33.726642 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.726596    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk" podUID="25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" containerName="authorino" containerID="cri-o://54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41" gracePeriod=30
Jul 17 05:02:33.975218 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:33.975186    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 05:02:34.017468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.017436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr2wj\" (UniqueName: \"kubernetes.io/projected/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-kube-api-access-zr2wj\") pod \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") "
Jul 17 05:02:34.017635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.017564    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-tls-cert\") pod \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\" (UID: \"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60\") "
Jul 17 05:02:34.020180 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.020144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-kube-api-access-zr2wj" (OuterVolumeSpecName: "kube-api-access-zr2wj") pod "25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" (UID: "25caeaa6-00b8-4c9b-9729-f2d5bd42ab60"). InnerVolumeSpecName "kube-api-access-zr2wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:02:34.027633 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.027601    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" (UID: "25caeaa6-00b8-4c9b-9729-f2d5bd42ab60"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:02:34.118696 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.118596    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-tls-cert\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:02:34.118696 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.118640    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2wj\" (UniqueName: \"kubernetes.io/projected/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60-kube-api-access-zr2wj\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:02:34.689658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.689627    2582 generic.go:356] "Generic (PLEG): container finished" podID="25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" containerID="54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41" exitCode=0
Jul 17 05:02:34.689658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.689673    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk"
Jul 17 05:02:34.690178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.689708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk" event={"ID":"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60","Type":"ContainerDied","Data":"54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41"}
Jul 17 05:02:34.690178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.689755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dcb66dbcc-5vbnk" event={"ID":"25caeaa6-00b8-4c9b-9729-f2d5bd42ab60","Type":"ContainerDied","Data":"ce97293771130ac26f7530435f120ecccf907d8668930e81cf82a9647cc37e97"}
Jul 17 05:02:34.690178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.689780    2582 scope.go:117] "RemoveContainer" containerID="54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41"
Jul 17 05:02:34.699016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.698996    2582 scope.go:117] "RemoveContainer" containerID="54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41"
Jul 17 05:02:34.699362 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:02:34.699336    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41\": container with ID starting with 54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41 not found: ID does not exist" containerID="54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41"
Jul 17 05:02:34.699483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.699365    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41"} err="failed to get container status \"54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41\": rpc error: code = NotFound desc = could not find container \"54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41\": container with ID starting with 54374d1f35963026665993411e460e9b3821855a85d5be0ff698abbc5c444d41 not found: ID does not exist"
Jul 17 05:02:34.710804 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.710771    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dcb66dbcc-5vbnk"]
Jul 17 05:02:34.714423 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.714389    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dcb66dbcc-5vbnk"]
Jul 17 05:02:34.729884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:02:34.729857    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" path="/var/lib/kubelet/pods/25caeaa6-00b8-4c9b-9729-f2d5bd42ab60/volumes"
Jul 17 05:04:56.682939 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:04:56.682863    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:04:56.683680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:04:56.683652    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:09:56.711738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:09:56.711707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:09:56.712550 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:09:56.712534    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:14:56.738609 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:14:56.738580    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:14:56.739172 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:14:56.739136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:16:40.026742 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.026703    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"]
Jul 17 05:16:40.027179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.027008    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" containerName="authorino"
Jul 17 05:16:40.027179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.027020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" containerName="authorino"
Jul 17 05:16:40.027179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.027075    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="25caeaa6-00b8-4c9b-9729-f2d5bd42ab60" containerName="authorino"
Jul 17 05:16:40.029823 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.029806    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:16:40.033430 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.033404    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:16:40.033569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.033411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:16:40.039577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.039549    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"]
Jul 17 05:16:40.201792 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.201757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10e13a-6232-4994-9d8b-ea9803e3a8f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-dg87p\" (UID: \"fb10e13a-6232-4994-9d8b-ea9803e3a8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:16:40.303307 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.303200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10e13a-6232-4994-9d8b-ea9803e3a8f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-dg87p\" (UID: \"fb10e13a-6232-4994-9d8b-ea9803e3a8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:16:40.303931 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.303909    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10e13a-6232-4994-9d8b-ea9803e3a8f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-dg87p\" (UID: \"fb10e13a-6232-4994-9d8b-ea9803e3a8f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:16:40.341331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.341295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:16:40.475149 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:40.475112    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"]
Jul 17 05:16:40.478208 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:16:40.478176    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb10e13a_6232_4994_9d8b_ea9803e3a8f4.slice/crio-6bda59e008611fa7b3a8d94faec1f79ed818e284414db6cd18570e9204eebe55 WatchSource:0}: Error finding container 6bda59e008611fa7b3a8d94faec1f79ed818e284414db6cd18570e9204eebe55: Status 404 returned error can't find the container with id 6bda59e008611fa7b3a8d94faec1f79ed818e284414db6cd18570e9204eebe55
Jul 17 05:16:41.190719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:41.190686    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerID="2ba49d7759a0977c83c74977359840adef7d82cbad2531bc586e06e5608015b6" exitCode=7
Jul 17 05:16:41.191240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:41.190751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" event={"ID":"fb10e13a-6232-4994-9d8b-ea9803e3a8f4","Type":"ContainerDied","Data":"2ba49d7759a0977c83c74977359840adef7d82cbad2531bc586e06e5608015b6"}
Jul 17 05:16:41.191240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:41.190796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" event={"ID":"fb10e13a-6232-4994-9d8b-ea9803e3a8f4","Type":"ContainerStarted","Data":"6bda59e008611fa7b3a8d94faec1f79ed818e284414db6cd18570e9204eebe55"}
Jul 17 05:16:41.191240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:41.190965    2582 scope.go:117] "RemoveContainer" containerID="2ba49d7759a0977c83c74977359840adef7d82cbad2531bc586e06e5608015b6"
Jul 17 05:16:42.197022 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:42.196987    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerID="86401c96c906cab88b13c1cb7ee68847ccc2eaa582623a893e795d633060b22a" exitCode=7
Jul 17 05:16:42.197486 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:42.197041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" event={"ID":"fb10e13a-6232-4994-9d8b-ea9803e3a8f4","Type":"ContainerDied","Data":"86401c96c906cab88b13c1cb7ee68847ccc2eaa582623a893e795d633060b22a"}
Jul 17 05:16:42.197486 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:42.197092    2582 scope.go:117] "RemoveContainer" containerID="2ba49d7759a0977c83c74977359840adef7d82cbad2531bc586e06e5608015b6"
Jul 17 05:16:42.197486 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:42.197294    2582 scope.go:117] "RemoveContainer" containerID="86401c96c906cab88b13c1cb7ee68847ccc2eaa582623a893e795d633060b22a"
Jul 17 05:16:42.197595 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:16:42.197523    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-dg87p_odh-ai-gateway-infra(fb10e13a-6232-4994-9d8b-ea9803e3a8f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4"
Jul 17 05:16:43.203187 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:43.203159    2582 scope.go:117] "RemoveContainer" containerID="86401c96c906cab88b13c1cb7ee68847ccc2eaa582623a893e795d633060b22a"
Jul 17 05:16:43.203595 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:16:43.203381    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-dg87p_odh-ai-gateway-infra(fb10e13a-6232-4994-9d8b-ea9803e3a8f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4"
Jul 17 05:16:56.726592 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:56.726561    2582 scope.go:117] "RemoveContainer" containerID="86401c96c906cab88b13c1cb7ee68847ccc2eaa582623a893e795d633060b22a"
Jul 17 05:16:57.261781 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:57.261696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" event={"ID":"fb10e13a-6232-4994-9d8b-ea9803e3a8f4","Type":"ContainerStarted","Data":"393b182cab05e56aab2c87b72a0812724b57c0432dc725a5daf70d2c6f818d65"}
Jul 17 05:16:57.277538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:57.277492    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" podStartSLOduration=17.27747613 podStartE2EDuration="17.27747613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:16:57.275856331 +0000 UTC m=+1621.012918680" watchObservedRunningTime="2026-07-17 05:16:57.27747613 +0000 UTC m=+1621.014538465"
Jul 17 05:16:58.268100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:58.268064    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerID="393b182cab05e56aab2c87b72a0812724b57c0432dc725a5daf70d2c6f818d65" exitCode=7
Jul 17 05:16:58.268538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:58.268110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" event={"ID":"fb10e13a-6232-4994-9d8b-ea9803e3a8f4","Type":"ContainerDied","Data":"393b182cab05e56aab2c87b72a0812724b57c0432dc725a5daf70d2c6f818d65"}
Jul 17 05:16:58.268538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:58.268184    2582 scope.go:117] "RemoveContainer" containerID="86401c96c906cab88b13c1cb7ee68847ccc2eaa582623a893e795d633060b22a"
Jul 17 05:16:58.268538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:58.268517    2582 scope.go:117] "RemoveContainer" containerID="393b182cab05e56aab2c87b72a0812724b57c0432dc725a5daf70d2c6f818d65"
Jul 17 05:16:58.268752 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:16:58.268726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-dg87p_odh-ai-gateway-infra(fb10e13a-6232-4994-9d8b-ea9803e3a8f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4"
Jul 17 05:16:58.285179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:58.285153    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"]
Jul 17 05:16:59.406458 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:59.406429    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:16:59.452034 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:59.452000    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10e13a-6232-4994-9d8b-ea9803e3a8f4-maas-api-service-ca\") pod \"fb10e13a-6232-4994-9d8b-ea9803e3a8f4\" (UID: \"fb10e13a-6232-4994-9d8b-ea9803e3a8f4\") "
Jul 17 05:16:59.452478 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:59.452446    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb10e13a-6232-4994-9d8b-ea9803e3a8f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb10e13a-6232-4994-9d8b-ea9803e3a8f4" (UID: "fb10e13a-6232-4994-9d8b-ea9803e3a8f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:16:59.552965 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:16:59.552875    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10e13a-6232-4994-9d8b-ea9803e3a8f4-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:17:00.278750 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:00.278717    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"
Jul 17 05:17:00.278925 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:00.278717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p" event={"ID":"fb10e13a-6232-4994-9d8b-ea9803e3a8f4","Type":"ContainerDied","Data":"6bda59e008611fa7b3a8d94faec1f79ed818e284414db6cd18570e9204eebe55"}
Jul 17 05:17:00.278925 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:00.278864    2582 scope.go:117] "RemoveContainer" containerID="393b182cab05e56aab2c87b72a0812724b57c0432dc725a5daf70d2c6f818d65"
Jul 17 05:17:00.299195 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:00.299163    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"]
Jul 17 05:17:00.304185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:00.304152    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-dg87p"]
Jul 17 05:17:00.729438 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:00.729405    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" path="/var/lib/kubelet/pods/fb10e13a-6232-4994-9d8b-ea9803e3a8f4/volumes"
Jul 17 05:17:40.069181 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069144    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"]
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069465    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069476    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069494    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069499    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069551    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069559    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.069673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.069565    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:17:40.072455 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.072438    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:17:40.075012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.074974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:17:40.075143 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.075016    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:17:40.079716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.079608    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"]
Jul 17 05:17:40.190727 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.190682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c905af5-48b0-47f9-bbe9-b61f17e38aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-kfnl2\" (UID: \"7c905af5-48b0-47f9-bbe9-b61f17e38aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:17:40.292078 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.292041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c905af5-48b0-47f9-bbe9-b61f17e38aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-kfnl2\" (UID: \"7c905af5-48b0-47f9-bbe9-b61f17e38aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:17:40.292738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.292719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c905af5-48b0-47f9-bbe9-b61f17e38aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-kfnl2\" (UID: \"7c905af5-48b0-47f9-bbe9-b61f17e38aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:17:40.384593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.384500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:17:40.517178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:40.517148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"]
Jul 17 05:17:40.519517 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:17:40.519489    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c905af5_48b0_47f9_bbe9_b61f17e38aba.slice/crio-7c0df11303cce3a0f3c1cefd18be13e4c25e9087a2c3f7530be27f21896861f1 WatchSource:0}: Error finding container 7c0df11303cce3a0f3c1cefd18be13e4c25e9087a2c3f7530be27f21896861f1: Status 404 returned error can't find the container with id 7c0df11303cce3a0f3c1cefd18be13e4c25e9087a2c3f7530be27f21896861f1
Jul 17 05:17:41.452235 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:41.452198    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerID="3286a4071921fbd3ed8c521f6e0c202975fdf987b4d924ec75d027b1d02bc1ed" exitCode=7
Jul 17 05:17:41.452716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:41.452292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" event={"ID":"7c905af5-48b0-47f9-bbe9-b61f17e38aba","Type":"ContainerDied","Data":"3286a4071921fbd3ed8c521f6e0c202975fdf987b4d924ec75d027b1d02bc1ed"}
Jul 17 05:17:41.452716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:41.452354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" event={"ID":"7c905af5-48b0-47f9-bbe9-b61f17e38aba","Type":"ContainerStarted","Data":"7c0df11303cce3a0f3c1cefd18be13e4c25e9087a2c3f7530be27f21896861f1"}
Jul 17 05:17:41.452716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:41.452497    2582 scope.go:117] "RemoveContainer" containerID="3286a4071921fbd3ed8c521f6e0c202975fdf987b4d924ec75d027b1d02bc1ed"
Jul 17 05:17:42.458841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:42.458810    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerID="cbfe8e70eb07c8ee68f930dc77ab0b3e4e555be29a3a22cd79b13cea7cb992b6" exitCode=7
Jul 17 05:17:42.459289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:42.458852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" event={"ID":"7c905af5-48b0-47f9-bbe9-b61f17e38aba","Type":"ContainerDied","Data":"cbfe8e70eb07c8ee68f930dc77ab0b3e4e555be29a3a22cd79b13cea7cb992b6"}
Jul 17 05:17:42.459289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:42.458906    2582 scope.go:117] "RemoveContainer" containerID="3286a4071921fbd3ed8c521f6e0c202975fdf987b4d924ec75d027b1d02bc1ed"
Jul 17 05:17:42.459289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:42.459087    2582 scope.go:117] "RemoveContainer" containerID="cbfe8e70eb07c8ee68f930dc77ab0b3e4e555be29a3a22cd79b13cea7cb992b6"
Jul 17 05:17:42.459404 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:17:42.459330    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-kfnl2_odh-ai-gateway-infra(7c905af5-48b0-47f9-bbe9-b61f17e38aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba"
Jul 17 05:17:43.464295 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:43.464247    2582 scope.go:117] "RemoveContainer" containerID="cbfe8e70eb07c8ee68f930dc77ab0b3e4e555be29a3a22cd79b13cea7cb992b6"
Jul 17 05:17:43.464691 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:17:43.464461    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-kfnl2_odh-ai-gateway-infra(7c905af5-48b0-47f9-bbe9-b61f17e38aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba"
Jul 17 05:17:58.725459 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:58.725417    2582 scope.go:117] "RemoveContainer" containerID="cbfe8e70eb07c8ee68f930dc77ab0b3e4e555be29a3a22cd79b13cea7cb992b6"
Jul 17 05:17:59.531693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:59.531662    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerID="e28635c3bd0f4c90945c55871811cc33237aa183e0f3f1682f418099305d9f29" exitCode=7
Jul 17 05:17:59.531868 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:59.531712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" event={"ID":"7c905af5-48b0-47f9-bbe9-b61f17e38aba","Type":"ContainerDied","Data":"e28635c3bd0f4c90945c55871811cc33237aa183e0f3f1682f418099305d9f29"}
Jul 17 05:17:59.531868 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:59.531761    2582 scope.go:117] "RemoveContainer" containerID="cbfe8e70eb07c8ee68f930dc77ab0b3e4e555be29a3a22cd79b13cea7cb992b6"
Jul 17 05:17:59.532050 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:17:59.532031    2582 scope.go:117] "RemoveContainer" containerID="e28635c3bd0f4c90945c55871811cc33237aa183e0f3f1682f418099305d9f29"
Jul 17 05:17:59.532247 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:17:59.532229    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-kfnl2_odh-ai-gateway-infra(7c905af5-48b0-47f9-bbe9-b61f17e38aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba"
Jul 17 05:18:00.557352 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:00.557313    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"]
Jul 17 05:18:00.689696 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:00.689671    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:18:00.758776 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:00.758737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c905af5-48b0-47f9-bbe9-b61f17e38aba-maas-api-service-ca\") pod \"7c905af5-48b0-47f9-bbe9-b61f17e38aba\" (UID: \"7c905af5-48b0-47f9-bbe9-b61f17e38aba\") "
Jul 17 05:18:00.759149 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:00.759120    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c905af5-48b0-47f9-bbe9-b61f17e38aba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c905af5-48b0-47f9-bbe9-b61f17e38aba" (UID: "7c905af5-48b0-47f9-bbe9-b61f17e38aba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:18:00.859530 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:00.859440    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c905af5-48b0-47f9-bbe9-b61f17e38aba-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:18:01.543469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:01.543433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2" event={"ID":"7c905af5-48b0-47f9-bbe9-b61f17e38aba","Type":"ContainerDied","Data":"7c0df11303cce3a0f3c1cefd18be13e4c25e9087a2c3f7530be27f21896861f1"}
Jul 17 05:18:01.543469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:01.543457    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"
Jul 17 05:18:01.543750 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:01.543510    2582 scope.go:117] "RemoveContainer" containerID="e28635c3bd0f4c90945c55871811cc33237aa183e0f3f1682f418099305d9f29"
Jul 17 05:18:01.564699 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:01.564662    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"]
Jul 17 05:18:01.572549 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:01.572517    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kfnl2"]
Jul 17 05:18:02.731573 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:02.731536    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" path="/var/lib/kubelet/pods/7c905af5-48b0-47f9-bbe9-b61f17e38aba/volumes"
Jul 17 05:18:40.115830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.115786    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"]
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116193    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116212    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116226    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116234    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb10e13a-6232-4994-9d8b-ea9803e3a8f4" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116282    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116291    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116371    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116382    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.116481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.116393    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:18:40.119436 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.119413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:40.122164 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.122139    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:18:40.122164 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.122154    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:18:40.127736 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.127712    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"]
Jul 17 05:18:40.189789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.189748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529c9160-7bda-4391-90b4-6c8294caab6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ffr9k\" (UID: \"529c9160-7bda-4391-90b4-6c8294caab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:40.290783 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.290740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529c9160-7bda-4391-90b4-6c8294caab6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ffr9k\" (UID: \"529c9160-7bda-4391-90b4-6c8294caab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:40.291452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.291429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529c9160-7bda-4391-90b4-6c8294caab6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ffr9k\" (UID: \"529c9160-7bda-4391-90b4-6c8294caab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:40.431656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.431553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:40.572214 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.572089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"]
Jul 17 05:18:40.574858 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:18:40.574829    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529c9160_7bda_4391_90b4_6c8294caab6f.slice/crio-b0b77966b0c1b077b67f7a49d1c796dbdae532443363ce310ec575f3234c40e8 WatchSource:0}: Error finding container b0b77966b0c1b077b67f7a49d1c796dbdae532443363ce310ec575f3234c40e8: Status 404 returned error can't find the container with id b0b77966b0c1b077b67f7a49d1c796dbdae532443363ce310ec575f3234c40e8
Jul 17 05:18:40.720843 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:40.720809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" event={"ID":"529c9160-7bda-4391-90b4-6c8294caab6f","Type":"ContainerStarted","Data":"b0b77966b0c1b077b67f7a49d1c796dbdae532443363ce310ec575f3234c40e8"}
Jul 17 05:18:41.726508 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:41.726471    2582 generic.go:356] "Generic (PLEG): container finished" podID="529c9160-7bda-4391-90b4-6c8294caab6f" containerID="77de51beb461a1f5b72e67719120ec5551ab84c30ea200455b09f5779377e09e" exitCode=7
Jul 17 05:18:41.726954 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:41.726540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" event={"ID":"529c9160-7bda-4391-90b4-6c8294caab6f","Type":"ContainerDied","Data":"77de51beb461a1f5b72e67719120ec5551ab84c30ea200455b09f5779377e09e"}
Jul 17 05:18:41.726954 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:41.726742    2582 scope.go:117] "RemoveContainer" containerID="77de51beb461a1f5b72e67719120ec5551ab84c30ea200455b09f5779377e09e"
Jul 17 05:18:42.734710 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:42.734672    2582 generic.go:356] "Generic (PLEG): container finished" podID="529c9160-7bda-4391-90b4-6c8294caab6f" containerID="e4c3b1429f5e314ed33e200bdb9f5c0204d99b897c27b55c170aa0db10be1f70" exitCode=7
Jul 17 05:18:42.735308 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:42.735279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" event={"ID":"529c9160-7bda-4391-90b4-6c8294caab6f","Type":"ContainerDied","Data":"e4c3b1429f5e314ed33e200bdb9f5c0204d99b897c27b55c170aa0db10be1f70"}
Jul 17 05:18:42.735468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:42.735453    2582 scope.go:117] "RemoveContainer" containerID="77de51beb461a1f5b72e67719120ec5551ab84c30ea200455b09f5779377e09e"
Jul 17 05:18:42.736392 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:42.736368    2582 scope.go:117] "RemoveContainer" containerID="e4c3b1429f5e314ed33e200bdb9f5c0204d99b897c27b55c170aa0db10be1f70"
Jul 17 05:18:42.736837 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:18:42.736812    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ffr9k_odh-ai-gateway-infra(529c9160-7bda-4391-90b4-6c8294caab6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" podUID="529c9160-7bda-4391-90b4-6c8294caab6f"
Jul 17 05:18:43.740794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:43.740762    2582 scope.go:117] "RemoveContainer" containerID="e4c3b1429f5e314ed33e200bdb9f5c0204d99b897c27b55c170aa0db10be1f70"
Jul 17 05:18:43.741299 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:18:43.740994    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ffr9k_odh-ai-gateway-infra(529c9160-7bda-4391-90b4-6c8294caab6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" podUID="529c9160-7bda-4391-90b4-6c8294caab6f"
Jul 17 05:18:56.726521 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:56.726484    2582 scope.go:117] "RemoveContainer" containerID="e4c3b1429f5e314ed33e200bdb9f5c0204d99b897c27b55c170aa0db10be1f70"
Jul 17 05:18:57.800431 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:57.800398    2582 generic.go:356] "Generic (PLEG): container finished" podID="529c9160-7bda-4391-90b4-6c8294caab6f" containerID="ea1dcbfa0de7b14065c01ec7b13c16e36cb6c3aa4b35d4e2de92983fc6f82b2b" exitCode=7
Jul 17 05:18:57.800840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:57.800471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" event={"ID":"529c9160-7bda-4391-90b4-6c8294caab6f","Type":"ContainerDied","Data":"ea1dcbfa0de7b14065c01ec7b13c16e36cb6c3aa4b35d4e2de92983fc6f82b2b"}
Jul 17 05:18:57.800840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:57.800535    2582 scope.go:117] "RemoveContainer" containerID="e4c3b1429f5e314ed33e200bdb9f5c0204d99b897c27b55c170aa0db10be1f70"
Jul 17 05:18:57.800952 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:57.800929    2582 scope.go:117] "RemoveContainer" containerID="ea1dcbfa0de7b14065c01ec7b13c16e36cb6c3aa4b35d4e2de92983fc6f82b2b"
Jul 17 05:18:57.801175 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:18:57.801155    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ffr9k_odh-ai-gateway-infra(529c9160-7bda-4391-90b4-6c8294caab6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" podUID="529c9160-7bda-4391-90b4-6c8294caab6f"
Jul 17 05:18:58.824978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:58.824939    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"]
Jul 17 05:18:58.958301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:58.958234    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:59.022917 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.022884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529c9160-7bda-4391-90b4-6c8294caab6f-maas-api-service-ca\") pod \"529c9160-7bda-4391-90b4-6c8294caab6f\" (UID: \"529c9160-7bda-4391-90b4-6c8294caab6f\") "
Jul 17 05:18:59.023234 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.023211    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529c9160-7bda-4391-90b4-6c8294caab6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "529c9160-7bda-4391-90b4-6c8294caab6f" (UID: "529c9160-7bda-4391-90b4-6c8294caab6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:18:59.124378 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.124286    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529c9160-7bda-4391-90b4-6c8294caab6f-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:18:59.813038 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.813003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k" event={"ID":"529c9160-7bda-4391-90b4-6c8294caab6f","Type":"ContainerDied","Data":"b0b77966b0c1b077b67f7a49d1c796dbdae532443363ce310ec575f3234c40e8"}
Jul 17 05:18:59.813038 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.813023    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"
Jul 17 05:18:59.813315 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.813065    2582 scope.go:117] "RemoveContainer" containerID="ea1dcbfa0de7b14065c01ec7b13c16e36cb6c3aa4b35d4e2de92983fc6f82b2b"
Jul 17 05:18:59.835887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.835852    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"]
Jul 17 05:18:59.840141 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:18:59.840114    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ffr9k"]
Jul 17 05:19:00.730623 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:00.730585    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" path="/var/lib/kubelet/pods/529c9160-7bda-4391-90b4-6c8294caab6f/volumes"
Jul 17 05:19:30.158081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158043    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"]
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158341    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158352    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158365    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158371    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158384    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158390    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c905af5-48b0-47f9-bbe9-b61f17e38aba" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158432    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158438    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.158651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.158444    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:30.161401 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.161384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:30.163817 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.163789    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:19:30.163961 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.163874    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:19:30.168087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.168061    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"]
Jul 17 05:19:30.290412 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.290366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eec8882-246f-40b3-a323-4fed4e14e9a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6rqhx\" (UID: \"7eec8882-246f-40b3-a323-4fed4e14e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:30.391715 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.391660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eec8882-246f-40b3-a323-4fed4e14e9a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6rqhx\" (UID: \"7eec8882-246f-40b3-a323-4fed4e14e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:30.392344 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.392326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eec8882-246f-40b3-a323-4fed4e14e9a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6rqhx\" (UID: \"7eec8882-246f-40b3-a323-4fed4e14e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:30.472545 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.472511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:30.606397 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.606371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"]
Jul 17 05:19:30.608831 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:19:30.608800    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eec8882_246f_40b3_a323_4fed4e14e9a9.slice/crio-32bf37fc6b1c6549498c7dbe796dd5e9ba51539d1db800dc180f2663dc48aa9f WatchSource:0}: Error finding container 32bf37fc6b1c6549498c7dbe796dd5e9ba51539d1db800dc180f2663dc48aa9f: Status 404 returned error can't find the container with id 32bf37fc6b1c6549498c7dbe796dd5e9ba51539d1db800dc180f2663dc48aa9f
Jul 17 05:19:30.941372 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.941339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerStarted","Data":"6018a8ecae11de1787185b18645d5dadc6322c5e9b9648d45591854c4b437fd8"}
Jul 17 05:19:30.941543 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.941388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerStarted","Data":"32bf37fc6b1c6549498c7dbe796dd5e9ba51539d1db800dc180f2663dc48aa9f"}
Jul 17 05:19:30.957643 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:30.957588    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" podStartSLOduration=0.957570926 podStartE2EDuration="957.570926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:19:30.955860122 +0000 UTC m=+1774.692922459" watchObservedRunningTime="2026-07-17 05:19:30.957570926 +0000 UTC m=+1774.694633262"
Jul 17 05:19:31.946889 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:31.946851    2582 generic.go:356] "Generic (PLEG): container finished" podID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerID="6018a8ecae11de1787185b18645d5dadc6322c5e9b9648d45591854c4b437fd8" exitCode=7
Jul 17 05:19:31.947313 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:31.946908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerDied","Data":"6018a8ecae11de1787185b18645d5dadc6322c5e9b9648d45591854c4b437fd8"}
Jul 17 05:19:31.947313 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:31.947164    2582 scope.go:117] "RemoveContainer" containerID="6018a8ecae11de1787185b18645d5dadc6322c5e9b9648d45591854c4b437fd8"
Jul 17 05:19:32.953437 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:32.953400    2582 generic.go:356] "Generic (PLEG): container finished" podID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerID="6dfd4798981279f88bd4d659b96755a7270a5cd3271085d7ab4de4eba419a9bc" exitCode=7
Jul 17 05:19:32.953890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:32.953443    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerDied","Data":"6dfd4798981279f88bd4d659b96755a7270a5cd3271085d7ab4de4eba419a9bc"}
Jul 17 05:19:32.953890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:32.953506    2582 scope.go:117] "RemoveContainer" containerID="6018a8ecae11de1787185b18645d5dadc6322c5e9b9648d45591854c4b437fd8"
Jul 17 05:19:32.954021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:32.953997    2582 scope.go:117] "RemoveContainer" containerID="6dfd4798981279f88bd4d659b96755a7270a5cd3271085d7ab4de4eba419a9bc"
Jul 17 05:19:32.959756 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:19:32.954405    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6rqhx_odh-ai-gateway-infra(7eec8882-246f-40b3-a323-4fed4e14e9a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9"
Jul 17 05:19:33.959597 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:33.959571    2582 scope.go:117] "RemoveContainer" containerID="6dfd4798981279f88bd4d659b96755a7270a5cd3271085d7ab4de4eba419a9bc"
Jul 17 05:19:33.959986 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:19:33.959746    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6rqhx_odh-ai-gateway-infra(7eec8882-246f-40b3-a323-4fed4e14e9a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9"
Jul 17 05:19:47.724667 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:47.724580    2582 scope.go:117] "RemoveContainer" containerID="6dfd4798981279f88bd4d659b96755a7270a5cd3271085d7ab4de4eba419a9bc"
Jul 17 05:19:48.018870 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:48.018787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerStarted","Data":"95352eb87c1423a617d371da5c13bf2dd3d7896758c5f6cd5e194f412d965c91"}
Jul 17 05:19:49.024852 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:49.024822    2582 generic.go:356] "Generic (PLEG): container finished" podID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerID="95352eb87c1423a617d371da5c13bf2dd3d7896758c5f6cd5e194f412d965c91" exitCode=6
Jul 17 05:19:49.025342 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:49.024863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerDied","Data":"95352eb87c1423a617d371da5c13bf2dd3d7896758c5f6cd5e194f412d965c91"}
Jul 17 05:19:49.025342 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:49.024940    2582 scope.go:117] "RemoveContainer" containerID="6dfd4798981279f88bd4d659b96755a7270a5cd3271085d7ab4de4eba419a9bc"
Jul 17 05:19:49.025342 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:49.025180    2582 scope.go:117] "RemoveContainer" containerID="95352eb87c1423a617d371da5c13bf2dd3d7896758c5f6cd5e194f412d965c91"
Jul 17 05:19:49.025500 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:19:49.025418    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6rqhx_odh-ai-gateway-infra(7eec8882-246f-40b3-a323-4fed4e14e9a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9"
Jul 17 05:19:49.047252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:49.047225    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"]
Jul 17 05:19:50.160171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:50.160144    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:50.260787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:50.260750    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eec8882-246f-40b3-a323-4fed4e14e9a9-maas-api-service-ca\") pod \"7eec8882-246f-40b3-a323-4fed4e14e9a9\" (UID: \"7eec8882-246f-40b3-a323-4fed4e14e9a9\") "
Jul 17 05:19:50.261098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:50.261076    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eec8882-246f-40b3-a323-4fed4e14e9a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7eec8882-246f-40b3-a323-4fed4e14e9a9" (UID: "7eec8882-246f-40b3-a323-4fed4e14e9a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:19:50.361634 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:50.361552    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eec8882-246f-40b3-a323-4fed4e14e9a9-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:19:51.035931 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:51.035840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx" event={"ID":"7eec8882-246f-40b3-a323-4fed4e14e9a9","Type":"ContainerDied","Data":"32bf37fc6b1c6549498c7dbe796dd5e9ba51539d1db800dc180f2663dc48aa9f"}
Jul 17 05:19:51.035931 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:51.035907    2582 scope.go:117] "RemoveContainer" containerID="95352eb87c1423a617d371da5c13bf2dd3d7896758c5f6cd5e194f412d965c91"
Jul 17 05:19:51.036154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:51.035858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"
Jul 17 05:19:51.052085 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:51.052053    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"]
Jul 17 05:19:51.058329 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:51.058299    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rqhx"]
Jul 17 05:19:52.729919 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:52.729885    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" path="/var/lib/kubelet/pods/7eec8882-246f-40b3-a323-4fed4e14e9a9/volumes"
Jul 17 05:19:56.761037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:56.761009    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:19:56.763593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:56.763570    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:19:58.322772 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.322738    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"]
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323042    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323053    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323065    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323071    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323081    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323086    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="529c9160-7bda-4391-90b4-6c8294caab6f" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323153    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.323157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323159    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.323414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.323165    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:19:58.327393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.327375    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:19:58.329972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.329954    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:19:58.330074 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.329956    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:19:58.333206 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.333178    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"]
Jul 17 05:19:58.432688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.432651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a73240-3aa4-4bde-95cb-2a38c95f68e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx\" (UID: \"12a73240-3aa4-4bde-95cb-2a38c95f68e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:19:58.533616 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.533576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a73240-3aa4-4bde-95cb-2a38c95f68e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx\" (UID: \"12a73240-3aa4-4bde-95cb-2a38c95f68e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:19:58.534210 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.534189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a73240-3aa4-4bde-95cb-2a38c95f68e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx\" (UID: \"12a73240-3aa4-4bde-95cb-2a38c95f68e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:19:58.639150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.639073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:19:58.772376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:58.772345    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"]
Jul 17 05:19:58.775677 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:19:58.775653    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a73240_3aa4_4bde_95cb_2a38c95f68e4.slice/crio-51d7ebe73c6bb592449c57d0037994210788fa63c50a597a3431bb802276bb63 WatchSource:0}: Error finding container 51d7ebe73c6bb592449c57d0037994210788fa63c50a597a3431bb802276bb63: Status 404 returned error can't find the container with id 51d7ebe73c6bb592449c57d0037994210788fa63c50a597a3431bb802276bb63
Jul 17 05:19:59.071039 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:59.071005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerStarted","Data":"94f4f5d1ff48a5d702a8d315484be1cb9d72c7e34f44e344d97c5aa48e6528f0"}
Jul 17 05:19:59.071212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:59.071054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerStarted","Data":"51d7ebe73c6bb592449c57d0037994210788fa63c50a597a3431bb802276bb63"}
Jul 17 05:19:59.091608 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:19:59.091530    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" podStartSLOduration=1.091512172 podStartE2EDuration="1.091512172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:19:59.090141306 +0000 UTC m=+1802.827203643" watchObservedRunningTime="2026-07-17 05:19:59.091512172 +0000 UTC m=+1802.828574510"
Jul 17 05:20:00.076574 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:00.076535    2582 generic.go:356] "Generic (PLEG): container finished" podID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerID="94f4f5d1ff48a5d702a8d315484be1cb9d72c7e34f44e344d97c5aa48e6528f0" exitCode=7
Jul 17 05:20:00.077067 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:00.076593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerDied","Data":"94f4f5d1ff48a5d702a8d315484be1cb9d72c7e34f44e344d97c5aa48e6528f0"}
Jul 17 05:20:00.077067 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:00.076852    2582 scope.go:117] "RemoveContainer" containerID="94f4f5d1ff48a5d702a8d315484be1cb9d72c7e34f44e344d97c5aa48e6528f0"
Jul 17 05:20:01.082634 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:01.082602    2582 generic.go:356] "Generic (PLEG): container finished" podID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerID="d0c4c8642608edff4ba0caf61335ff51e0583b92bb7d0839339709e34bd8f421" exitCode=7
Jul 17 05:20:01.083098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:01.082687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerDied","Data":"d0c4c8642608edff4ba0caf61335ff51e0583b92bb7d0839339709e34bd8f421"}
Jul 17 05:20:01.083098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:01.082758    2582 scope.go:117] "RemoveContainer" containerID="94f4f5d1ff48a5d702a8d315484be1cb9d72c7e34f44e344d97c5aa48e6528f0"
Jul 17 05:20:01.083098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:01.082882    2582 scope.go:117] "RemoveContainer" containerID="d0c4c8642608edff4ba0caf61335ff51e0583b92bb7d0839339709e34bd8f421"
Jul 17 05:20:01.083098 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:01.083071    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx_odh-ai-gateway-infra(12a73240-3aa4-4bde-95cb-2a38c95f68e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4"
Jul 17 05:20:02.089555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:02.089528    2582 scope.go:117] "RemoveContainer" containerID="d0c4c8642608edff4ba0caf61335ff51e0583b92bb7d0839339709e34bd8f421"
Jul 17 05:20:02.090040 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:02.089698    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx_odh-ai-gateway-infra(12a73240-3aa4-4bde-95cb-2a38c95f68e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4"
Jul 17 05:20:13.724963 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:13.724931    2582 scope.go:117] "RemoveContainer" containerID="d0c4c8642608edff4ba0caf61335ff51e0583b92bb7d0839339709e34bd8f421"
Jul 17 05:20:14.144641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:14.144605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerStarted","Data":"9db5c136caf6be2c211e97840d7ab17b3d285b649ba04d69279fc03d920e459b"}
Jul 17 05:20:15.150840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:15.150800    2582 generic.go:356] "Generic (PLEG): container finished" podID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerID="9db5c136caf6be2c211e97840d7ab17b3d285b649ba04d69279fc03d920e459b" exitCode=7
Jul 17 05:20:15.151321 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:15.150870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerDied","Data":"9db5c136caf6be2c211e97840d7ab17b3d285b649ba04d69279fc03d920e459b"}
Jul 17 05:20:15.151321 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:15.150933    2582 scope.go:117] "RemoveContainer" containerID="d0c4c8642608edff4ba0caf61335ff51e0583b92bb7d0839339709e34bd8f421"
Jul 17 05:20:15.151321 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:15.151276    2582 scope.go:117] "RemoveContainer" containerID="9db5c136caf6be2c211e97840d7ab17b3d285b649ba04d69279fc03d920e459b"
Jul 17 05:20:15.151517 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:15.151495    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx_odh-ai-gateway-infra(12a73240-3aa4-4bde-95cb-2a38c95f68e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4"
Jul 17 05:20:15.170514 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:15.170480    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"]
Jul 17 05:20:16.290621 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:16.290595    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:20:16.397573 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:16.397532    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a73240-3aa4-4bde-95cb-2a38c95f68e4-maas-api-service-ca\") pod \"12a73240-3aa4-4bde-95cb-2a38c95f68e4\" (UID: \"12a73240-3aa4-4bde-95cb-2a38c95f68e4\") "
Jul 17 05:20:16.397900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:16.397877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a73240-3aa4-4bde-95cb-2a38c95f68e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12a73240-3aa4-4bde-95cb-2a38c95f68e4" (UID: "12a73240-3aa4-4bde-95cb-2a38c95f68e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:20:16.498410 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:16.498374    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a73240-3aa4-4bde-95cb-2a38c95f68e4-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:20:17.162290 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:17.162239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx" event={"ID":"12a73240-3aa4-4bde-95cb-2a38c95f68e4","Type":"ContainerDied","Data":"51d7ebe73c6bb592449c57d0037994210788fa63c50a597a3431bb802276bb63"}
Jul 17 05:20:17.162477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:17.162348    2582 scope.go:117] "RemoveContainer" containerID="9db5c136caf6be2c211e97840d7ab17b3d285b649ba04d69279fc03d920e459b"
Jul 17 05:20:17.162477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:17.162267    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"
Jul 17 05:20:17.178562 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:17.178531    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"]
Jul 17 05:20:17.183834 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:17.183805    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-hzqdx"]
Jul 17 05:20:18.730136 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:18.730102    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" path="/var/lib/kubelet/pods/12a73240-3aa4-4bde-95cb-2a38c95f68e4/volumes"
Jul 17 05:20:30.206486 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206449    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"]
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206775    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206787    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206802    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206808    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eec8882-246f-40b3-a323-4fed4e14e9a9" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206814    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206820    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206867    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206875    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.206890 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.206882    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:30.209631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.209613    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:30.212052 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.212029    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:20:30.212183 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.212055    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:20:30.217146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.216878    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"]
Jul 17 05:20:30.314815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.314775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12978940-f10c-4560-b885-a13432118bb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-msqcf\" (UID: \"12978940-f10c-4560-b885-a13432118bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:30.415951 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.415914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12978940-f10c-4560-b885-a13432118bb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-msqcf\" (UID: \"12978940-f10c-4560-b885-a13432118bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:30.416584 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.416565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12978940-f10c-4560-b885-a13432118bb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-msqcf\" (UID: \"12978940-f10c-4560-b885-a13432118bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:30.520873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.520787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:30.660175 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:30.660140    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"]
Jul 17 05:20:30.663818 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:20:30.663787    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12978940_f10c_4560_b885_a13432118bb3.slice/crio-4ac0290e5d22887ffe6e307d213f79dcddd94338a7e8da777c588630b182e796 WatchSource:0}: Error finding container 4ac0290e5d22887ffe6e307d213f79dcddd94338a7e8da777c588630b182e796: Status 404 returned error can't find the container with id 4ac0290e5d22887ffe6e307d213f79dcddd94338a7e8da777c588630b182e796
Jul 17 05:20:31.221159 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:31.221123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerStarted","Data":"e4ca3694acd0a7d4fc8986bb0557aeb1deab3542ef56e2da2a2da957b39ba1af"}
Jul 17 05:20:31.221590 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:31.221179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerStarted","Data":"4ac0290e5d22887ffe6e307d213f79dcddd94338a7e8da777c588630b182e796"}
Jul 17 05:20:31.237797 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:31.237744    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" podStartSLOduration=1.237725562 podStartE2EDuration="1.237725562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:20:31.235528788 +0000 UTC m=+1834.972591139" watchObservedRunningTime="2026-07-17 05:20:31.237725562 +0000 UTC m=+1834.974787897"
Jul 17 05:20:32.227212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:32.227176    2582 generic.go:356] "Generic (PLEG): container finished" podID="12978940-f10c-4560-b885-a13432118bb3" containerID="e4ca3694acd0a7d4fc8986bb0557aeb1deab3542ef56e2da2a2da957b39ba1af" exitCode=6
Jul 17 05:20:32.227675 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:32.227285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerDied","Data":"e4ca3694acd0a7d4fc8986bb0557aeb1deab3542ef56e2da2a2da957b39ba1af"}
Jul 17 05:20:32.227675 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:32.227475    2582 scope.go:117] "RemoveContainer" containerID="e4ca3694acd0a7d4fc8986bb0557aeb1deab3542ef56e2da2a2da957b39ba1af"
Jul 17 05:20:33.233192 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:33.233158    2582 generic.go:356] "Generic (PLEG): container finished" podID="12978940-f10c-4560-b885-a13432118bb3" containerID="4ea9ff6cda3199e33d3aa19e556c306b89eda12b2bcb75aaac0f6c8c0cf8a2ed" exitCode=6
Jul 17 05:20:33.233812 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:33.233223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerDied","Data":"4ea9ff6cda3199e33d3aa19e556c306b89eda12b2bcb75aaac0f6c8c0cf8a2ed"}
Jul 17 05:20:33.233812 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:33.233292    2582 scope.go:117] "RemoveContainer" containerID="e4ca3694acd0a7d4fc8986bb0557aeb1deab3542ef56e2da2a2da957b39ba1af"
Jul 17 05:20:33.233812 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:33.233498    2582 scope.go:117] "RemoveContainer" containerID="4ea9ff6cda3199e33d3aa19e556c306b89eda12b2bcb75aaac0f6c8c0cf8a2ed"
Jul 17 05:20:33.233812 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:33.233741    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-msqcf_odh-ai-gateway-infra(12978940-f10c-4560-b885-a13432118bb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" podUID="12978940-f10c-4560-b885-a13432118bb3"
Jul 17 05:20:34.239697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:34.239672    2582 scope.go:117] "RemoveContainer" containerID="4ea9ff6cda3199e33d3aa19e556c306b89eda12b2bcb75aaac0f6c8c0cf8a2ed"
Jul 17 05:20:34.240117 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:34.239876    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-msqcf_odh-ai-gateway-infra(12978940-f10c-4560-b885-a13432118bb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" podUID="12978940-f10c-4560-b885-a13432118bb3"
Jul 17 05:20:44.724796 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:44.724760    2582 scope.go:117] "RemoveContainer" containerID="4ea9ff6cda3199e33d3aa19e556c306b89eda12b2bcb75aaac0f6c8c0cf8a2ed"
Jul 17 05:20:45.288188 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:45.288161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerStarted","Data":"b021db2eda0f45fdfeea66c9bea47942f369b5badfe7cb7da5ce96606256c1fa"}
Jul 17 05:20:46.294304 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:46.294254    2582 generic.go:356] "Generic (PLEG): container finished" podID="12978940-f10c-4560-b885-a13432118bb3" containerID="b021db2eda0f45fdfeea66c9bea47942f369b5badfe7cb7da5ce96606256c1fa" exitCode=6
Jul 17 05:20:46.294304 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:46.294300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerDied","Data":"b021db2eda0f45fdfeea66c9bea47942f369b5badfe7cb7da5ce96606256c1fa"}
Jul 17 05:20:46.294764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:46.294365    2582 scope.go:117] "RemoveContainer" containerID="4ea9ff6cda3199e33d3aa19e556c306b89eda12b2bcb75aaac0f6c8c0cf8a2ed"
Jul 17 05:20:46.294764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:46.294658    2582 scope.go:117] "RemoveContainer" containerID="b021db2eda0f45fdfeea66c9bea47942f369b5badfe7cb7da5ce96606256c1fa"
Jul 17 05:20:46.294866 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:46.294846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-msqcf_odh-ai-gateway-infra(12978940-f10c-4560-b885-a13432118bb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" podUID="12978940-f10c-4560-b885-a13432118bb3"
Jul 17 05:20:46.318752 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:46.318719    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"]
Jul 17 05:20:47.430319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:47.430294    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:47.453746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:47.453712    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12978940-f10c-4560-b885-a13432118bb3-maas-api-service-ca\") pod \"12978940-f10c-4560-b885-a13432118bb3\" (UID: \"12978940-f10c-4560-b885-a13432118bb3\") "
Jul 17 05:20:47.454053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:47.454032    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12978940-f10c-4560-b885-a13432118bb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12978940-f10c-4560-b885-a13432118bb3" (UID: "12978940-f10c-4560-b885-a13432118bb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:20:47.560077 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:47.555550    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12978940-f10c-4560-b885-a13432118bb3-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:20:48.306349 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.306311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf" event={"ID":"12978940-f10c-4560-b885-a13432118bb3","Type":"ContainerDied","Data":"4ac0290e5d22887ffe6e307d213f79dcddd94338a7e8da777c588630b182e796"}
Jul 17 05:20:48.306534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.306371    2582 scope.go:117] "RemoveContainer" containerID="b021db2eda0f45fdfeea66c9bea47942f369b5badfe7cb7da5ce96606256c1fa"
Jul 17 05:20:48.306534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.306320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"
Jul 17 05:20:48.329598 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.329568    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"]
Jul 17 05:20:48.335843 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.335815    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-msqcf"]
Jul 17 05:20:48.362567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362537    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"]
Jul 17 05:20:48.362829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362814    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362832    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a73240-3aa4-4bde-95cb-2a38c95f68e4" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362843    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362849    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362858    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362864    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362870    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.362894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362875    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.363101 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362931    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.363101 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362939    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.363101 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.362946    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12978940-f10c-4560-b885-a13432118bb3" containerName="revoke-keys"
Jul 17 05:20:48.367131 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.367113    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:20:48.370029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.369837    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:20:48.370029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.369969    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:20:48.371905 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.371881    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"]
Jul 17 05:20:48.463644 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.463604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg\" (UID: \"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:20:48.564683 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.564591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg\" (UID: \"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:20:48.565200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.565180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg\" (UID: \"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:20:48.680050 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.679996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:20:48.730997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.730952    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12978940-f10c-4560-b885-a13432118bb3" path="/var/lib/kubelet/pods/12978940-f10c-4560-b885-a13432118bb3/volumes"
Jul 17 05:20:48.815093 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:48.815017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"]
Jul 17 05:20:48.818139 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:20:48.818109    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a36060_39f0_4ee9_9bc0_6aa6aed3c9d1.slice/crio-8288983bed4aaeaeb4f3ceeeb252ef077e437825a78cdc48691784e0483173f6 WatchSource:0}: Error finding container 8288983bed4aaeaeb4f3ceeeb252ef077e437825a78cdc48691784e0483173f6: Status 404 returned error can't find the container with id 8288983bed4aaeaeb4f3ceeeb252ef077e437825a78cdc48691784e0483173f6
Jul 17 05:20:49.312364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:49.312329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" event={"ID":"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1","Type":"ContainerStarted","Data":"3184fee3cf53309dce053cfb86e2ee21d2d694242920a6634adc31d591f37bdb"}
Jul 17 05:20:49.312543 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:49.312378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" event={"ID":"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1","Type":"ContainerStarted","Data":"8288983bed4aaeaeb4f3ceeeb252ef077e437825a78cdc48691784e0483173f6"}
Jul 17 05:20:49.329866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:49.329810    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" podStartSLOduration=1.329794169 podStartE2EDuration="1.329794169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:20:49.327659413 +0000 UTC m=+1853.064721749" watchObservedRunningTime="2026-07-17 05:20:49.329794169 +0000 UTC m=+1853.066856539"
Jul 17 05:20:50.318884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:50.318851    2582 generic.go:356] "Generic (PLEG): container finished" podID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerID="3184fee3cf53309dce053cfb86e2ee21d2d694242920a6634adc31d591f37bdb" exitCode=7
Jul 17 05:20:50.319319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:50.318909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" event={"ID":"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1","Type":"ContainerDied","Data":"3184fee3cf53309dce053cfb86e2ee21d2d694242920a6634adc31d591f37bdb"}
Jul 17 05:20:50.319319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:50.319083    2582 scope.go:117] "RemoveContainer" containerID="3184fee3cf53309dce053cfb86e2ee21d2d694242920a6634adc31d591f37bdb"
Jul 17 05:20:51.325825 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:51.325792    2582 generic.go:356] "Generic (PLEG): container finished" podID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerID="6b5cf7cca14b4a414707f5830fbb4fba56ca0de9120cdac98f416261f836de25" exitCode=7
Jul 17 05:20:51.326285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:51.325849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" event={"ID":"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1","Type":"ContainerDied","Data":"6b5cf7cca14b4a414707f5830fbb4fba56ca0de9120cdac98f416261f836de25"}
Jul 17 05:20:51.326285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:51.325898    2582 scope.go:117] "RemoveContainer" containerID="3184fee3cf53309dce053cfb86e2ee21d2d694242920a6634adc31d591f37bdb"
Jul 17 05:20:51.326285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:51.326095    2582 scope.go:117] "RemoveContainer" containerID="6b5cf7cca14b4a414707f5830fbb4fba56ca0de9120cdac98f416261f836de25"
Jul 17 05:20:51.326416 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:51.326330    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg_odh-ai-gateway-infra(64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1"
Jul 17 05:20:52.331492 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:20:52.331460    2582 scope.go:117] "RemoveContainer" containerID="6b5cf7cca14b4a414707f5830fbb4fba56ca0de9120cdac98f416261f836de25"
Jul 17 05:20:52.331868 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:20:52.331672    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg_odh-ai-gateway-infra(64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1"
Jul 17 05:21:06.728060 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:06.728033    2582 scope.go:117] "RemoveContainer" containerID="6b5cf7cca14b4a414707f5830fbb4fba56ca0de9120cdac98f416261f836de25"
Jul 17 05:21:07.398195 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:07.398161    2582 generic.go:356] "Generic (PLEG): container finished" podID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerID="55917ddc17322c89ea246fcce99a7b663d046572e8dc73d790e276d3998141f0" exitCode=6
Jul 17 05:21:07.398345 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:07.398193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" event={"ID":"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1","Type":"ContainerDied","Data":"55917ddc17322c89ea246fcce99a7b663d046572e8dc73d790e276d3998141f0"}
Jul 17 05:21:07.398345 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:07.398285    2582 scope.go:117] "RemoveContainer" containerID="6b5cf7cca14b4a414707f5830fbb4fba56ca0de9120cdac98f416261f836de25"
Jul 17 05:21:07.398568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:07.398553    2582 scope.go:117] "RemoveContainer" containerID="55917ddc17322c89ea246fcce99a7b663d046572e8dc73d790e276d3998141f0"
Jul 17 05:21:07.398769 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:07.398753    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg_odh-ai-gateway-infra(64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1"
Jul 17 05:21:08.424627 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:08.424590    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"]
Jul 17 05:21:08.558350 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:08.558325    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:21:08.631571 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:08.631536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1-maas-api-service-ca\") pod \"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1\" (UID: \"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1\") "
Jul 17 05:21:08.631909 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:08.631887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" (UID: "64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:21:08.732693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:08.732664    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:21:09.408387 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:09.408345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg" event={"ID":"64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1","Type":"ContainerDied","Data":"8288983bed4aaeaeb4f3ceeeb252ef077e437825a78cdc48691784e0483173f6"}
Jul 17 05:21:09.408587 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:09.408411    2582 scope.go:117] "RemoveContainer" containerID="55917ddc17322c89ea246fcce99a7b663d046572e8dc73d790e276d3998141f0"
Jul 17 05:21:09.408587 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:09.408359    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"
Jul 17 05:21:09.425025 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:09.424994    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"]
Jul 17 05:21:09.430926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:09.430895    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-8gbqg"]
Jul 17 05:21:10.729787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:10.729752    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" path="/var/lib/kubelet/pods/64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1/volumes"
Jul 17 05:21:16.510631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.510594    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"]
Jul 17 05:21:16.511120 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.510996    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511120 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511012    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511120 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511034    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511120 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511042    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511120 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511054    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511120 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511062    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511458 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511134    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511458 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511145    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.511458 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.511157    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a36060-39f0-4ee9-9bc0-6aa6aed3c9d1" containerName="revoke-keys"
Jul 17 05:21:16.515593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.515570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:16.517947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.517924    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:21:16.518067 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.517992    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:21:16.521952 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.521925    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"]
Jul 17 05:21:16.597282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.597218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d994227e-6f63-4fc1-bcb0-7bd06572a579-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh\" (UID: \"d994227e-6f63-4fc1-bcb0-7bd06572a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:16.698012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.697976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d994227e-6f63-4fc1-bcb0-7bd06572a579-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh\" (UID: \"d994227e-6f63-4fc1-bcb0-7bd06572a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:16.698626 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.698609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d994227e-6f63-4fc1-bcb0-7bd06572a579-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh\" (UID: \"d994227e-6f63-4fc1-bcb0-7bd06572a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:16.826940 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.826850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:16.960207 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:16.960174    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"]
Jul 17 05:21:16.962939 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:21:16.962910    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd994227e_6f63_4fc1_bcb0_7bd06572a579.slice/crio-2e19b2fc26188d1615f2952ae84778c152959e3fb6eaa657823d7a9d8304e7b4 WatchSource:0}: Error finding container 2e19b2fc26188d1615f2952ae84778c152959e3fb6eaa657823d7a9d8304e7b4: Status 404 returned error can't find the container with id 2e19b2fc26188d1615f2952ae84778c152959e3fb6eaa657823d7a9d8304e7b4
Jul 17 05:21:17.441661 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:17.441631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" event={"ID":"d994227e-6f63-4fc1-bcb0-7bd06572a579","Type":"ContainerStarted","Data":"e909cbf1568c161d3e5d4de41322b79fb4194d8b53b7ade78731476f81074351"}
Jul 17 05:21:17.441825 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:17.441679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" event={"ID":"d994227e-6f63-4fc1-bcb0-7bd06572a579","Type":"ContainerStarted","Data":"2e19b2fc26188d1615f2952ae84778c152959e3fb6eaa657823d7a9d8304e7b4"}
Jul 17 05:21:17.456084 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:17.456036    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" podStartSLOduration=1.456020229 podStartE2EDuration="1.456020229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:21:17.455164049 +0000 UTC m=+1881.192226385" watchObservedRunningTime="2026-07-17 05:21:17.456020229 +0000 UTC m=+1881.193082565"
Jul 17 05:21:18.447548 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:18.447518    2582 generic.go:356] "Generic (PLEG): container finished" podID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerID="e909cbf1568c161d3e5d4de41322b79fb4194d8b53b7ade78731476f81074351" exitCode=7
Jul 17 05:21:18.447911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:18.447554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" event={"ID":"d994227e-6f63-4fc1-bcb0-7bd06572a579","Type":"ContainerDied","Data":"e909cbf1568c161d3e5d4de41322b79fb4194d8b53b7ade78731476f81074351"}
Jul 17 05:21:18.447911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:18.447742    2582 scope.go:117] "RemoveContainer" containerID="e909cbf1568c161d3e5d4de41322b79fb4194d8b53b7ade78731476f81074351"
Jul 17 05:21:19.453650 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:19.453616    2582 generic.go:356] "Generic (PLEG): container finished" podID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerID="02075895d379fcffea2fc049fc9823d84b222f38e0659914917a35d3b816c87d" exitCode=7
Jul 17 05:21:19.454066 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:19.453672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" event={"ID":"d994227e-6f63-4fc1-bcb0-7bd06572a579","Type":"ContainerDied","Data":"02075895d379fcffea2fc049fc9823d84b222f38e0659914917a35d3b816c87d"}
Jul 17 05:21:19.454066 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:19.453776    2582 scope.go:117] "RemoveContainer" containerID="e909cbf1568c161d3e5d4de41322b79fb4194d8b53b7ade78731476f81074351"
Jul 17 05:21:19.454066 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:19.453963    2582 scope.go:117] "RemoveContainer" containerID="02075895d379fcffea2fc049fc9823d84b222f38e0659914917a35d3b816c87d"
Jul 17 05:21:19.454233 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:19.454210    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh_odh-ai-gateway-infra(d994227e-6f63-4fc1-bcb0-7bd06572a579)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579"
Jul 17 05:21:20.248866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.248828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"]
Jul 17 05:21:20.253153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.253134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:20.260778 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.260465    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"]
Jul 17 05:21:20.330214 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.330170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ca1db-65af-448d-bfc6-853f725015ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-n9slx\" (UID: \"c17ca1db-65af-448d-bfc6-853f725015ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:20.431127 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.431082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ca1db-65af-448d-bfc6-853f725015ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-n9slx\" (UID: \"c17ca1db-65af-448d-bfc6-853f725015ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:20.431765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.431742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ca1db-65af-448d-bfc6-853f725015ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-n9slx\" (UID: \"c17ca1db-65af-448d-bfc6-853f725015ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:20.462359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.462335    2582 scope.go:117] "RemoveContainer" containerID="02075895d379fcffea2fc049fc9823d84b222f38e0659914917a35d3b816c87d"
Jul 17 05:21:20.462733 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:20.462512    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh_odh-ai-gateway-infra(d994227e-6f63-4fc1-bcb0-7bd06572a579)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579"
Jul 17 05:21:20.564604 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.564515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:20.702776 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:20.702749    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"]
Jul 17 05:21:20.704459 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:21:20.704433    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17ca1db_65af_448d_bfc6_853f725015ad.slice/crio-c36015255f170671ef93882daf3a4b2f80f344769bb820eff461c49a4039cd6f WatchSource:0}: Error finding container c36015255f170671ef93882daf3a4b2f80f344769bb820eff461c49a4039cd6f: Status 404 returned error can't find the container with id c36015255f170671ef93882daf3a4b2f80f344769bb820eff461c49a4039cd6f
Jul 17 05:21:21.467965 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:21.467932    2582 generic.go:356] "Generic (PLEG): container finished" podID="c17ca1db-65af-448d-bfc6-853f725015ad" containerID="6493842b59027d3230f2eada63cca138b7233a843a93087fb032cf34548042df" exitCode=6
Jul 17 05:21:21.468376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:21.468025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" event={"ID":"c17ca1db-65af-448d-bfc6-853f725015ad","Type":"ContainerDied","Data":"6493842b59027d3230f2eada63cca138b7233a843a93087fb032cf34548042df"}
Jul 17 05:21:21.468376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:21.468074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" event={"ID":"c17ca1db-65af-448d-bfc6-853f725015ad","Type":"ContainerStarted","Data":"c36015255f170671ef93882daf3a4b2f80f344769bb820eff461c49a4039cd6f"}
Jul 17 05:21:21.468376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:21.468196    2582 scope.go:117] "RemoveContainer" containerID="6493842b59027d3230f2eada63cca138b7233a843a93087fb032cf34548042df"
Jul 17 05:21:22.110542 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:22.110502    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17ca1db_65af_448d_bfc6_853f725015ad.slice/crio-859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17ca1db_65af_448d_bfc6_853f725015ad.slice/crio-conmon-859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 05:21:22.474821 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:22.474787    2582 generic.go:356] "Generic (PLEG): container finished" podID="c17ca1db-65af-448d-bfc6-853f725015ad" containerID="859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31" exitCode=6
Jul 17 05:21:22.475240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:22.474868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" event={"ID":"c17ca1db-65af-448d-bfc6-853f725015ad","Type":"ContainerDied","Data":"859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31"}
Jul 17 05:21:22.475240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:22.474926    2582 scope.go:117] "RemoveContainer" containerID="6493842b59027d3230f2eada63cca138b7233a843a93087fb032cf34548042df"
Jul 17 05:21:22.475240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:22.475151    2582 scope.go:117] "RemoveContainer" containerID="859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31"
Jul 17 05:21:22.475410 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:22.475389    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-n9slx_odh-ai-gateway-infra(c17ca1db-65af-448d-bfc6-853f725015ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" podUID="c17ca1db-65af-448d-bfc6-853f725015ad"
Jul 17 05:21:23.481077 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:23.481049    2582 scope.go:117] "RemoveContainer" containerID="859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31"
Jul 17 05:21:23.481499 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:23.481244    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-n9slx_odh-ai-gateway-infra(c17ca1db-65af-448d-bfc6-853f725015ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" podUID="c17ca1db-65af-448d-bfc6-853f725015ad"
Jul 17 05:21:33.724713 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:33.724678    2582 scope.go:117] "RemoveContainer" containerID="859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31"
Jul 17 05:21:34.528974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:34.528941    2582 generic.go:356] "Generic (PLEG): container finished" podID="c17ca1db-65af-448d-bfc6-853f725015ad" containerID="6360e0d8810d86c2e7be70ebf24471e191ec0d7111f9d1518350a9fdbe931e60" exitCode=6
Jul 17 05:21:34.529226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:34.529022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" event={"ID":"c17ca1db-65af-448d-bfc6-853f725015ad","Type":"ContainerDied","Data":"6360e0d8810d86c2e7be70ebf24471e191ec0d7111f9d1518350a9fdbe931e60"}
Jul 17 05:21:34.529226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:34.529093    2582 scope.go:117] "RemoveContainer" containerID="859644c085203890730ae86398e2e4d390b118d598fd4fd4c04a105694e72b31"
Jul 17 05:21:34.529463 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:34.529448    2582 scope.go:117] "RemoveContainer" containerID="6360e0d8810d86c2e7be70ebf24471e191ec0d7111f9d1518350a9fdbe931e60"
Jul 17 05:21:34.529687 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:34.529666    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-n9slx_odh-ai-gateway-infra(c17ca1db-65af-448d-bfc6-853f725015ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" podUID="c17ca1db-65af-448d-bfc6-853f725015ad"
Jul 17 05:21:34.725464 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:34.725438    2582 scope.go:117] "RemoveContainer" containerID="02075895d379fcffea2fc049fc9823d84b222f38e0659914917a35d3b816c87d"
Jul 17 05:21:35.535469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.535437    2582 generic.go:356] "Generic (PLEG): container finished" podID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerID="335b2278848b0be94d2fa14e21382cf7cb9be4dfd28d6535ddb958102d27775b" exitCode=7
Jul 17 05:21:35.535699 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.535506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" event={"ID":"d994227e-6f63-4fc1-bcb0-7bd06572a579","Type":"ContainerDied","Data":"335b2278848b0be94d2fa14e21382cf7cb9be4dfd28d6535ddb958102d27775b"}
Jul 17 05:21:35.535699 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.535567    2582 scope.go:117] "RemoveContainer" containerID="02075895d379fcffea2fc049fc9823d84b222f38e0659914917a35d3b816c87d"
Jul 17 05:21:35.535916 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.535899    2582 scope.go:117] "RemoveContainer" containerID="335b2278848b0be94d2fa14e21382cf7cb9be4dfd28d6535ddb958102d27775b"
Jul 17 05:21:35.536156 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:35.536136    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh_odh-ai-gateway-infra(d994227e-6f63-4fc1-bcb0-7bd06572a579)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579"
Jul 17 05:21:35.556327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.556297    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"]
Jul 17 05:21:35.688536 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.688513    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:35.755887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.755849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ca1db-65af-448d-bfc6-853f725015ad-maas-api-service-ca\") pod \"c17ca1db-65af-448d-bfc6-853f725015ad\" (UID: \"c17ca1db-65af-448d-bfc6-853f725015ad\") "
Jul 17 05:21:35.756297 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.756227    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17ca1db-65af-448d-bfc6-853f725015ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c17ca1db-65af-448d-bfc6-853f725015ad" (UID: "c17ca1db-65af-448d-bfc6-853f725015ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:21:35.857307 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:35.857196    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ca1db-65af-448d-bfc6-853f725015ad-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:21:36.542874 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.542847    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"
Jul 17 05:21:36.543153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.542890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx" event={"ID":"c17ca1db-65af-448d-bfc6-853f725015ad","Type":"ContainerDied","Data":"c36015255f170671ef93882daf3a4b2f80f344769bb820eff461c49a4039cd6f"}
Jul 17 05:21:36.543153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.542958    2582 scope.go:117] "RemoveContainer" containerID="6360e0d8810d86c2e7be70ebf24471e191ec0d7111f9d1518350a9fdbe931e60"
Jul 17 05:21:36.560100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.560068    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"]
Jul 17 05:21:36.564739 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.564702    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"]
Jul 17 05:21:36.569935 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.569906    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-n9slx"]
Jul 17 05:21:36.691046 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.691020    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:36.734802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.734757    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" path="/var/lib/kubelet/pods/c17ca1db-65af-448d-bfc6-853f725015ad/volumes"
Jul 17 05:21:36.764322 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.764287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d994227e-6f63-4fc1-bcb0-7bd06572a579-maas-api-service-ca\") pod \"d994227e-6f63-4fc1-bcb0-7bd06572a579\" (UID: \"d994227e-6f63-4fc1-bcb0-7bd06572a579\") "
Jul 17 05:21:36.764806 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.764735    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d994227e-6f63-4fc1-bcb0-7bd06572a579-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d994227e-6f63-4fc1-bcb0-7bd06572a579" (UID: "d994227e-6f63-4fc1-bcb0-7bd06572a579"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:21:36.865796 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:36.865708    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d994227e-6f63-4fc1-bcb0-7bd06572a579-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:21:37.551116 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:37.551086    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"
Jul 17 05:21:37.551359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:37.551125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh" event={"ID":"d994227e-6f63-4fc1-bcb0-7bd06572a579","Type":"ContainerDied","Data":"2e19b2fc26188d1615f2952ae84778c152959e3fb6eaa657823d7a9d8304e7b4"}
Jul 17 05:21:37.551359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:37.551196    2582 scope.go:117] "RemoveContainer" containerID="335b2278848b0be94d2fa14e21382cf7cb9be4dfd28d6535ddb958102d27775b"
Jul 17 05:21:37.571475 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:37.571440    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"]
Jul 17 05:21:37.576060 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:37.576035    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-fsjhh"]
Jul 17 05:21:38.729944 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:38.729910    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" path="/var/lib/kubelet/pods/d994227e-6f63-4fc1-bcb0-7bd06572a579/volumes"
Jul 17 05:21:48.426058 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426017    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"]
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426362    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426375    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426386    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426392    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426405    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426411    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426419    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426424    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426429    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426434    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426484    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426493    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426500    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.426555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.426510    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:21:48.429282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.429247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:21:48.431854 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.431831    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:21:48.431981 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.431876    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:21:48.437398 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.437373    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"]
Jul 17 05:21:48.463140 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.463105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d31858-cc28-410f-8a4e-208d67299c86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v\" (UID: \"82d31858-cc28-410f-8a4e-208d67299c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:21:48.563606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.563568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d31858-cc28-410f-8a4e-208d67299c86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v\" (UID: \"82d31858-cc28-410f-8a4e-208d67299c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:21:48.564205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.564182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d31858-cc28-410f-8a4e-208d67299c86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v\" (UID: \"82d31858-cc28-410f-8a4e-208d67299c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:21:48.740203 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.740170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:21:48.874795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:48.874759    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"]
Jul 17 05:21:48.878643 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:21:48.878619    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d31858_cc28_410f_8a4e_208d67299c86.slice/crio-462154e3a4a53e752336a51358a86eb114ea3f326e22af67b4cb00275bcf431a WatchSource:0}: Error finding container 462154e3a4a53e752336a51358a86eb114ea3f326e22af67b4cb00275bcf431a: Status 404 returned error can't find the container with id 462154e3a4a53e752336a51358a86eb114ea3f326e22af67b4cb00275bcf431a
Jul 17 05:21:49.605593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:49.605552    2582 generic.go:356] "Generic (PLEG): container finished" podID="82d31858-cc28-410f-8a4e-208d67299c86" containerID="e81087cec4d4d031c94770ee46e27680c069cb7ffdeb8f7c1074594ed83fe8ab" exitCode=6
Jul 17 05:21:49.605941 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:49.605602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" event={"ID":"82d31858-cc28-410f-8a4e-208d67299c86","Type":"ContainerDied","Data":"e81087cec4d4d031c94770ee46e27680c069cb7ffdeb8f7c1074594ed83fe8ab"}
Jul 17 05:21:49.605941 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:49.605687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" event={"ID":"82d31858-cc28-410f-8a4e-208d67299c86","Type":"ContainerStarted","Data":"462154e3a4a53e752336a51358a86eb114ea3f326e22af67b4cb00275bcf431a"}
Jul 17 05:21:49.605941 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:49.605871    2582 scope.go:117] "RemoveContainer" containerID="e81087cec4d4d031c94770ee46e27680c069cb7ffdeb8f7c1074594ed83fe8ab"
Jul 17 05:21:50.611900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:50.611815    2582 generic.go:356] "Generic (PLEG): container finished" podID="82d31858-cc28-410f-8a4e-208d67299c86" containerID="a3cd7b94f515a32d61b0ee010d1d4b4d08cf8aed6655ea7b71043e2be073f370" exitCode=6
Jul 17 05:21:50.611900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:50.611855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" event={"ID":"82d31858-cc28-410f-8a4e-208d67299c86","Type":"ContainerDied","Data":"a3cd7b94f515a32d61b0ee010d1d4b4d08cf8aed6655ea7b71043e2be073f370"}
Jul 17 05:21:50.611900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:50.611902    2582 scope.go:117] "RemoveContainer" containerID="e81087cec4d4d031c94770ee46e27680c069cb7ffdeb8f7c1074594ed83fe8ab"
Jul 17 05:21:50.612402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:50.612085    2582 scope.go:117] "RemoveContainer" containerID="a3cd7b94f515a32d61b0ee010d1d4b4d08cf8aed6655ea7b71043e2be073f370"
Jul 17 05:21:50.612402 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:50.612322    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v_odh-ai-gateway-infra(82d31858-cc28-410f-8a4e-208d67299c86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" podUID="82d31858-cc28-410f-8a4e-208d67299c86"
Jul 17 05:21:51.617770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:21:51.617743    2582 scope.go:117] "RemoveContainer" containerID="a3cd7b94f515a32d61b0ee010d1d4b4d08cf8aed6655ea7b71043e2be073f370"
Jul 17 05:21:51.618190 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:21:51.617953    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v_odh-ai-gateway-infra(82d31858-cc28-410f-8a4e-208d67299c86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" podUID="82d31858-cc28-410f-8a4e-208d67299c86"
Jul 17 05:22:06.726557 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:06.726526    2582 scope.go:117] "RemoveContainer" containerID="a3cd7b94f515a32d61b0ee010d1d4b4d08cf8aed6655ea7b71043e2be073f370"
Jul 17 05:22:07.685912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:07.685830    2582 generic.go:356] "Generic (PLEG): container finished" podID="82d31858-cc28-410f-8a4e-208d67299c86" containerID="0d614a8222f6c4480687ba10ef8315b2208f25881797aedb94f32883cf7a6ca5" exitCode=6
Jul 17 05:22:07.686094 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:07.685902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" event={"ID":"82d31858-cc28-410f-8a4e-208d67299c86","Type":"ContainerDied","Data":"0d614a8222f6c4480687ba10ef8315b2208f25881797aedb94f32883cf7a6ca5"}
Jul 17 05:22:07.686094 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:07.685967    2582 scope.go:117] "RemoveContainer" containerID="a3cd7b94f515a32d61b0ee010d1d4b4d08cf8aed6655ea7b71043e2be073f370"
Jul 17 05:22:07.686330 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:07.686299    2582 scope.go:117] "RemoveContainer" containerID="0d614a8222f6c4480687ba10ef8315b2208f25881797aedb94f32883cf7a6ca5"
Jul 17 05:22:07.686553 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:07.686532    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v_odh-ai-gateway-infra(82d31858-cc28-410f-8a4e-208d67299c86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" podUID="82d31858-cc28-410f-8a4e-208d67299c86"
Jul 17 05:22:08.712665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:08.712629    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"]
Jul 17 05:22:08.845510 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:08.845485    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:22:08.942417 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:08.942379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d31858-cc28-410f-8a4e-208d67299c86-maas-api-service-ca\") pod \"82d31858-cc28-410f-8a4e-208d67299c86\" (UID: \"82d31858-cc28-410f-8a4e-208d67299c86\") "
Jul 17 05:22:08.942751 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:08.942728    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d31858-cc28-410f-8a4e-208d67299c86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82d31858-cc28-410f-8a4e-208d67299c86" (UID: "82d31858-cc28-410f-8a4e-208d67299c86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:22:09.043816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:09.043720    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d31858-cc28-410f-8a4e-208d67299c86-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:22:09.696488 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:09.696449    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"
Jul 17 05:22:09.696649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:09.696457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v" event={"ID":"82d31858-cc28-410f-8a4e-208d67299c86","Type":"ContainerDied","Data":"462154e3a4a53e752336a51358a86eb114ea3f326e22af67b4cb00275bcf431a"}
Jul 17 05:22:09.696649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:09.696590    2582 scope.go:117] "RemoveContainer" containerID="0d614a8222f6c4480687ba10ef8315b2208f25881797aedb94f32883cf7a6ca5"
Jul 17 05:22:09.717344 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:09.717311    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"]
Jul 17 05:22:09.722834 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:09.722807    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wtk4v"]
Jul 17 05:22:10.297652 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.297609    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"]
Jul 17 05:22:10.298002 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.297984    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:10.298088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298005    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:10.298088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298019    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:22:10.298088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298027    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d994227e-6f63-4fc1-bcb0-7bd06572a579" containerName="revoke-keys"
Jul 17 05:22:10.298088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298041    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:10.298088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298050    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:10.298367 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298142    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17ca1db-65af-448d-bfc6-853f725015ad" containerName="revoke-keys"
Jul 17 05:22:10.298367 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298154    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:10.298367 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.298167    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:10.301203 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.301180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:10.303837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.303807    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:22:10.303960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.303814    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:22:10.308558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.308407    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"]
Jul 17 05:22:10.456075 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.456031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af811d9a-3d5d-495c-92f9-026ba183bfc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ksbjl\" (UID: \"af811d9a-3d5d-495c-92f9-026ba183bfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:10.557368 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.557239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af811d9a-3d5d-495c-92f9-026ba183bfc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ksbjl\" (UID: \"af811d9a-3d5d-495c-92f9-026ba183bfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:10.557926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.557899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af811d9a-3d5d-495c-92f9-026ba183bfc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ksbjl\" (UID: \"af811d9a-3d5d-495c-92f9-026ba183bfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:10.613203 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.613153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:10.730956 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.730918    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d31858-cc28-410f-8a4e-208d67299c86" path="/var/lib/kubelet/pods/82d31858-cc28-410f-8a4e-208d67299c86/volumes"
Jul 17 05:22:10.745889 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:10.745863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"]
Jul 17 05:22:10.748339 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:22:10.748314    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf811d9a_3d5d_495c_92f9_026ba183bfc7.slice/crio-48665277c0fbdd8c53066de1c06b539b6288936b4c000d545a5e0de643a9e2e3 WatchSource:0}: Error finding container 48665277c0fbdd8c53066de1c06b539b6288936b4c000d545a5e0de643a9e2e3: Status 404 returned error can't find the container with id 48665277c0fbdd8c53066de1c06b539b6288936b4c000d545a5e0de643a9e2e3
Jul 17 05:22:11.709146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:11.709112    2582 generic.go:356] "Generic (PLEG): container finished" podID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerID="560f1d19070b6296c7556a5bcc806b9c21c6db0d53b9f323ef5e88c0b6a999fd" exitCode=6
Jul 17 05:22:11.709335 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:11.709175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" event={"ID":"af811d9a-3d5d-495c-92f9-026ba183bfc7","Type":"ContainerDied","Data":"560f1d19070b6296c7556a5bcc806b9c21c6db0d53b9f323ef5e88c0b6a999fd"}
Jul 17 05:22:11.709335 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:11.709218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" event={"ID":"af811d9a-3d5d-495c-92f9-026ba183bfc7","Type":"ContainerStarted","Data":"48665277c0fbdd8c53066de1c06b539b6288936b4c000d545a5e0de643a9e2e3"}
Jul 17 05:22:11.709417 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:11.709405    2582 scope.go:117] "RemoveContainer" containerID="560f1d19070b6296c7556a5bcc806b9c21c6db0d53b9f323ef5e88c0b6a999fd"
Jul 17 05:22:12.714560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:12.714518    2582 generic.go:356] "Generic (PLEG): container finished" podID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerID="cef8cb174d61692bf7a7857a5d351543662d5b15bdcb56c7fd929a0acc8eb26b" exitCode=6
Jul 17 05:22:12.715000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:12.714593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" event={"ID":"af811d9a-3d5d-495c-92f9-026ba183bfc7","Type":"ContainerDied","Data":"cef8cb174d61692bf7a7857a5d351543662d5b15bdcb56c7fd929a0acc8eb26b"}
Jul 17 05:22:12.715000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:12.714658    2582 scope.go:117] "RemoveContainer" containerID="560f1d19070b6296c7556a5bcc806b9c21c6db0d53b9f323ef5e88c0b6a999fd"
Jul 17 05:22:12.715000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:12.714928    2582 scope.go:117] "RemoveContainer" containerID="cef8cb174d61692bf7a7857a5d351543662d5b15bdcb56c7fd929a0acc8eb26b"
Jul 17 05:22:12.715134 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:12.715108    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ksbjl_odh-ai-gateway-infra(af811d9a-3d5d-495c-92f9-026ba183bfc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7"
Jul 17 05:22:13.726759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:13.726730    2582 scope.go:117] "RemoveContainer" containerID="cef8cb174d61692bf7a7857a5d351543662d5b15bdcb56c7fd929a0acc8eb26b"
Jul 17 05:22:13.727127 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:13.726911    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ksbjl_odh-ai-gateway-infra(af811d9a-3d5d-495c-92f9-026ba183bfc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7"
Jul 17 05:22:16.551829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.551796    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"]
Jul 17 05:22:16.552226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.552087    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:16.552226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.552097    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:16.552226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.552157    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d31858-cc28-410f-8a4e-208d67299c86" containerName="revoke-keys"
Jul 17 05:22:16.555076 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.555058    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:16.561004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.560975    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"]
Jul 17 05:22:16.710493 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.710459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0463d1dd-874c-4520-b3b6-8436b3dacede-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88\" (UID: \"0463d1dd-874c-4520-b3b6-8436b3dacede\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:16.811691 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.811598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0463d1dd-874c-4520-b3b6-8436b3dacede-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88\" (UID: \"0463d1dd-874c-4520-b3b6-8436b3dacede\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:16.812319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.812294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0463d1dd-874c-4520-b3b6-8436b3dacede-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88\" (UID: \"0463d1dd-874c-4520-b3b6-8436b3dacede\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:16.867486 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:16.867451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:17.003707 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:17.003674    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"]
Jul 17 05:22:17.006642 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:22:17.006613    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0463d1dd_874c_4520_b3b6_8436b3dacede.slice/crio-998b4d320f4646ae8aa7a5d65d83eea411579d1477fa8ea7b276cc430635eb6b WatchSource:0}: Error finding container 998b4d320f4646ae8aa7a5d65d83eea411579d1477fa8ea7b276cc430635eb6b: Status 404 returned error can't find the container with id 998b4d320f4646ae8aa7a5d65d83eea411579d1477fa8ea7b276cc430635eb6b
Jul 17 05:22:17.743805 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:17.743771    2582 generic.go:356] "Generic (PLEG): container finished" podID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerID="c60ce1f625ddd8a7e04ceb6b16f994d68395bc4d5fd1794a856714db98a4e547" exitCode=7
Jul 17 05:22:17.744173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:17.743812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" event={"ID":"0463d1dd-874c-4520-b3b6-8436b3dacede","Type":"ContainerDied","Data":"c60ce1f625ddd8a7e04ceb6b16f994d68395bc4d5fd1794a856714db98a4e547"}
Jul 17 05:22:17.744173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:17.743851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" event={"ID":"0463d1dd-874c-4520-b3b6-8436b3dacede","Type":"ContainerStarted","Data":"998b4d320f4646ae8aa7a5d65d83eea411579d1477fa8ea7b276cc430635eb6b"}
Jul 17 05:22:17.744173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:17.744033    2582 scope.go:117] "RemoveContainer" containerID="c60ce1f625ddd8a7e04ceb6b16f994d68395bc4d5fd1794a856714db98a4e547"
Jul 17 05:22:18.749498 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:18.749452    2582 generic.go:356] "Generic (PLEG): container finished" podID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerID="83336cfd1f673fcddc714abb342e8d9da605db9d74e917b4f237419ca6feed76" exitCode=7
Jul 17 05:22:18.749974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:18.749512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" event={"ID":"0463d1dd-874c-4520-b3b6-8436b3dacede","Type":"ContainerDied","Data":"83336cfd1f673fcddc714abb342e8d9da605db9d74e917b4f237419ca6feed76"}
Jul 17 05:22:18.749974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:18.749571    2582 scope.go:117] "RemoveContainer" containerID="c60ce1f625ddd8a7e04ceb6b16f994d68395bc4d5fd1794a856714db98a4e547"
Jul 17 05:22:18.749974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:18.749739    2582 scope.go:117] "RemoveContainer" containerID="83336cfd1f673fcddc714abb342e8d9da605db9d74e917b4f237419ca6feed76"
Jul 17 05:22:18.750105 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:18.749987    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88_odh-ai-gateway-infra(0463d1dd-874c-4520-b3b6-8436b3dacede)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede"
Jul 17 05:22:19.755358 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:19.755324    2582 scope.go:117] "RemoveContainer" containerID="83336cfd1f673fcddc714abb342e8d9da605db9d74e917b4f237419ca6feed76"
Jul 17 05:22:19.755734 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:19.755546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88_odh-ai-gateway-infra(0463d1dd-874c-4520-b3b6-8436b3dacede)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede"
Jul 17 05:22:24.725559 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:24.725523    2582 scope.go:117] "RemoveContainer" containerID="cef8cb174d61692bf7a7857a5d351543662d5b15bdcb56c7fd929a0acc8eb26b"
Jul 17 05:22:25.782146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:25.782111    2582 generic.go:356] "Generic (PLEG): container finished" podID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerID="6addac91806552029d5e7d29c8ecd52432e72a7f8cfcb486769569e5d69f6b91" exitCode=6
Jul 17 05:22:25.782546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:25.782185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" event={"ID":"af811d9a-3d5d-495c-92f9-026ba183bfc7","Type":"ContainerDied","Data":"6addac91806552029d5e7d29c8ecd52432e72a7f8cfcb486769569e5d69f6b91"}
Jul 17 05:22:25.782546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:25.782288    2582 scope.go:117] "RemoveContainer" containerID="cef8cb174d61692bf7a7857a5d351543662d5b15bdcb56c7fd929a0acc8eb26b"
Jul 17 05:22:25.782546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:25.782538    2582 scope.go:117] "RemoveContainer" containerID="6addac91806552029d5e7d29c8ecd52432e72a7f8cfcb486769569e5d69f6b91"
Jul 17 05:22:25.782785 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:25.782765    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ksbjl_odh-ai-gateway-infra(af811d9a-3d5d-495c-92f9-026ba183bfc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7"
Jul 17 05:22:26.808866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:26.808831    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"]
Jul 17 05:22:26.945580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:26.945553    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:26.981321 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:26.981287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af811d9a-3d5d-495c-92f9-026ba183bfc7-maas-api-service-ca\") pod \"af811d9a-3d5d-495c-92f9-026ba183bfc7\" (UID: \"af811d9a-3d5d-495c-92f9-026ba183bfc7\") "
Jul 17 05:22:26.981631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:26.981611    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af811d9a-3d5d-495c-92f9-026ba183bfc7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af811d9a-3d5d-495c-92f9-026ba183bfc7" (UID: "af811d9a-3d5d-495c-92f9-026ba183bfc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:22:27.082418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:27.082334    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af811d9a-3d5d-495c-92f9-026ba183bfc7-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:22:27.796653 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:27.796621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl" event={"ID":"af811d9a-3d5d-495c-92f9-026ba183bfc7","Type":"ContainerDied","Data":"48665277c0fbdd8c53066de1c06b539b6288936b4c000d545a5e0de643a9e2e3"}
Jul 17 05:22:27.796813 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:27.796684    2582 scope.go:117] "RemoveContainer" containerID="6addac91806552029d5e7d29c8ecd52432e72a7f8cfcb486769569e5d69f6b91"
Jul 17 05:22:27.796813 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:27.796635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"
Jul 17 05:22:27.819052 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:27.819021    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"]
Jul 17 05:22:27.825247 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:27.825215    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ksbjl"]
Jul 17 05:22:28.729418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:28.729377    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" path="/var/lib/kubelet/pods/af811d9a-3d5d-495c-92f9-026ba183bfc7/volumes"
Jul 17 05:22:33.724996 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:33.724964    2582 scope.go:117] "RemoveContainer" containerID="83336cfd1f673fcddc714abb342e8d9da605db9d74e917b4f237419ca6feed76"
Jul 17 05:22:34.826782 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:34.826747    2582 generic.go:356] "Generic (PLEG): container finished" podID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerID="00ab27cef044de6182210117d187513deb233fe31ae43317592f8d3c94265f46" exitCode=7
Jul 17 05:22:34.827181 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:34.826824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" event={"ID":"0463d1dd-874c-4520-b3b6-8436b3dacede","Type":"ContainerDied","Data":"00ab27cef044de6182210117d187513deb233fe31ae43317592f8d3c94265f46"}
Jul 17 05:22:34.827181 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:34.826887    2582 scope.go:117] "RemoveContainer" containerID="83336cfd1f673fcddc714abb342e8d9da605db9d74e917b4f237419ca6feed76"
Jul 17 05:22:34.827331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:34.827220    2582 scope.go:117] "RemoveContainer" containerID="00ab27cef044de6182210117d187513deb233fe31ae43317592f8d3c94265f46"
Jul 17 05:22:34.827492 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:34.827469    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88_odh-ai-gateway-infra(0463d1dd-874c-4520-b3b6-8436b3dacede)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede"
Jul 17 05:22:35.851183 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:35.851148    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"]
Jul 17 05:22:35.983312 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:35.983288    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:36.046462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.046426    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0463d1dd-874c-4520-b3b6-8436b3dacede-maas-api-service-ca\") pod \"0463d1dd-874c-4520-b3b6-8436b3dacede\" (UID: \"0463d1dd-874c-4520-b3b6-8436b3dacede\") "
Jul 17 05:22:36.046811 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.046790    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0463d1dd-874c-4520-b3b6-8436b3dacede-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0463d1dd-874c-4520-b3b6-8436b3dacede" (UID: "0463d1dd-874c-4520-b3b6-8436b3dacede"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:22:36.147182 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.147102    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0463d1dd-874c-4520-b3b6-8436b3dacede-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:22:36.838608 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.838573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88" event={"ID":"0463d1dd-874c-4520-b3b6-8436b3dacede","Type":"ContainerDied","Data":"998b4d320f4646ae8aa7a5d65d83eea411579d1477fa8ea7b276cc430635eb6b"}
Jul 17 05:22:36.838785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.838629    2582 scope.go:117] "RemoveContainer" containerID="00ab27cef044de6182210117d187513deb233fe31ae43317592f8d3c94265f46"
Jul 17 05:22:36.838785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.838586    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"
Jul 17 05:22:36.853904 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.853875    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"]
Jul 17 05:22:36.857703 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:36.857672    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-v9b88"]
Jul 17 05:22:38.729797 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:38.729760    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" path="/var/lib/kubelet/pods/0463d1dd-874c-4520-b3b6-8436b3dacede/volumes"
Jul 17 05:22:48.479016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.478938    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"]
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479243    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479270    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479282    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479293    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479308    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479313    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479321    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479326    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479381    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479388    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:22:48.479391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479394    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:22:48.479737 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.479401    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:22:48.482742 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.482722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:22:48.485560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.485542    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:22:48.485648 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.485569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:22:48.491984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.491956    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"]
Jul 17 05:22:48.543021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.542976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8d635-d629-4eb9-b54f-7d0e1f423b06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z\" (UID: \"8dd8d635-d629-4eb9-b54f-7d0e1f423b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:22:48.643446 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.643418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8d635-d629-4eb9-b54f-7d0e1f423b06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z\" (UID: \"8dd8d635-d629-4eb9-b54f-7d0e1f423b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:22:48.644064 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.644043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8d635-d629-4eb9-b54f-7d0e1f423b06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z\" (UID: \"8dd8d635-d629-4eb9-b54f-7d0e1f423b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:22:48.794017 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.793929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:22:48.943342 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:48.943319    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"]
Jul 17 05:22:48.945929 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:22:48.945899    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd8d635_d629_4eb9_b54f_7d0e1f423b06.slice/crio-61dfeb484d62c7fddec373cdbad0dc4297d1d81f53820def19883cedd4b80948 WatchSource:0}: Error finding container 61dfeb484d62c7fddec373cdbad0dc4297d1d81f53820def19883cedd4b80948: Status 404 returned error can't find the container with id 61dfeb484d62c7fddec373cdbad0dc4297d1d81f53820def19883cedd4b80948
Jul 17 05:22:49.897445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:49.897356    2582 generic.go:356] "Generic (PLEG): container finished" podID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerID="f02073538a87d8e5465e92e806b081fa1b5c07cdfb4fc71180b8a3d9f9d10811" exitCode=6
Jul 17 05:22:49.897445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:49.897429    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" event={"ID":"8dd8d635-d629-4eb9-b54f-7d0e1f423b06","Type":"ContainerDied","Data":"f02073538a87d8e5465e92e806b081fa1b5c07cdfb4fc71180b8a3d9f9d10811"}
Jul 17 05:22:49.897857 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:49.897468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" event={"ID":"8dd8d635-d629-4eb9-b54f-7d0e1f423b06","Type":"ContainerStarted","Data":"61dfeb484d62c7fddec373cdbad0dc4297d1d81f53820def19883cedd4b80948"}
Jul 17 05:22:49.897857 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:49.897646    2582 scope.go:117] "RemoveContainer" containerID="f02073538a87d8e5465e92e806b081fa1b5c07cdfb4fc71180b8a3d9f9d10811"
Jul 17 05:22:50.902992 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:50.902895    2582 generic.go:356] "Generic (PLEG): container finished" podID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerID="93f9347998aaec49315a28a872a82b9b9d9bc8e7c3351ef04157733071ae873e" exitCode=6
Jul 17 05:22:50.902992 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:50.902958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" event={"ID":"8dd8d635-d629-4eb9-b54f-7d0e1f423b06","Type":"ContainerDied","Data":"93f9347998aaec49315a28a872a82b9b9d9bc8e7c3351ef04157733071ae873e"}
Jul 17 05:22:50.903494 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:50.903018    2582 scope.go:117] "RemoveContainer" containerID="f02073538a87d8e5465e92e806b081fa1b5c07cdfb4fc71180b8a3d9f9d10811"
Jul 17 05:22:50.903494 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:50.903193    2582 scope.go:117] "RemoveContainer" containerID="93f9347998aaec49315a28a872a82b9b9d9bc8e7c3351ef04157733071ae873e"
Jul 17 05:22:50.903494 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:50.903457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z_odh-ai-gateway-infra(8dd8d635-d629-4eb9-b54f-7d0e1f423b06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06"
Jul 17 05:22:51.908546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:22:51.908514    2582 scope.go:117] "RemoveContainer" containerID="93f9347998aaec49315a28a872a82b9b9d9bc8e7c3351ef04157733071ae873e"
Jul 17 05:22:51.908962 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:22:51.908733    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z_odh-ai-gateway-infra(8dd8d635-d629-4eb9-b54f-7d0e1f423b06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06"
Jul 17 05:23:00.343738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.343700    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"]
Jul 17 05:23:00.344215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.344113    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:23:00.344215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.344130    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:23:00.344215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.344144    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:23:00.344215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.344152    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:23:00.344472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.344253    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0463d1dd-874c-4520-b3b6-8436b3dacede" containerName="revoke-keys"
Jul 17 05:23:00.344472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.344291    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af811d9a-3d5d-495c-92f9-026ba183bfc7" containerName="revoke-keys"
Jul 17 05:23:00.347219 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.347195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:00.355518 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.355492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"]
Jul 17 05:23:00.433847 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.433808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a38b22e-8bdc-44e6-8fad-13ed6748d685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ks99l\" (UID: \"5a38b22e-8bdc-44e6-8fad-13ed6748d685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:00.534541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.534497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a38b22e-8bdc-44e6-8fad-13ed6748d685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ks99l\" (UID: \"5a38b22e-8bdc-44e6-8fad-13ed6748d685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:00.535152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.535132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a38b22e-8bdc-44e6-8fad-13ed6748d685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-ks99l\" (UID: \"5a38b22e-8bdc-44e6-8fad-13ed6748d685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:00.658124 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.658030    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:00.791414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.791388    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"]
Jul 17 05:23:00.793786 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:23:00.793757    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a38b22e_8bdc_44e6_8fad_13ed6748d685.slice/crio-e95abda1ff0c9ffbd6e5a596c429c798815ee2205f97aa388566eca7e8c2a390 WatchSource:0}: Error finding container e95abda1ff0c9ffbd6e5a596c429c798815ee2205f97aa388566eca7e8c2a390: Status 404 returned error can't find the container with id e95abda1ff0c9ffbd6e5a596c429c798815ee2205f97aa388566eca7e8c2a390
Jul 17 05:23:00.947003 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:00.946966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" event={"ID":"5a38b22e-8bdc-44e6-8fad-13ed6748d685","Type":"ContainerStarted","Data":"e95abda1ff0c9ffbd6e5a596c429c798815ee2205f97aa388566eca7e8c2a390"}
Jul 17 05:23:01.952866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:01.952830    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerID="a18619e646f422b9e2b1b168025fffe171e2445e5a200a9b46884dc22d9e840b" exitCode=6
Jul 17 05:23:01.953339 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:01.952900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" event={"ID":"5a38b22e-8bdc-44e6-8fad-13ed6748d685","Type":"ContainerDied","Data":"a18619e646f422b9e2b1b168025fffe171e2445e5a200a9b46884dc22d9e840b"}
Jul 17 05:23:01.953339 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:01.953087    2582 scope.go:117] "RemoveContainer" containerID="a18619e646f422b9e2b1b168025fffe171e2445e5a200a9b46884dc22d9e840b"
Jul 17 05:23:02.724980 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:02.724953    2582 scope.go:117] "RemoveContainer" containerID="93f9347998aaec49315a28a872a82b9b9d9bc8e7c3351ef04157733071ae873e"
Jul 17 05:23:02.958411 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:02.958381    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerID="0aee60fc705c361cc869c4f622eb09c188324407626715e8d8c4cdd7f0148827" exitCode=6
Jul 17 05:23:02.958881 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:02.958463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" event={"ID":"5a38b22e-8bdc-44e6-8fad-13ed6748d685","Type":"ContainerDied","Data":"0aee60fc705c361cc869c4f622eb09c188324407626715e8d8c4cdd7f0148827"}
Jul 17 05:23:02.958881 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:02.958528    2582 scope.go:117] "RemoveContainer" containerID="a18619e646f422b9e2b1b168025fffe171e2445e5a200a9b46884dc22d9e840b"
Jul 17 05:23:02.958881 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:02.958716    2582 scope.go:117] "RemoveContainer" containerID="0aee60fc705c361cc869c4f622eb09c188324407626715e8d8c4cdd7f0148827"
Jul 17 05:23:02.959038 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:02.958943    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ks99l_odh-ai-gateway-infra(5a38b22e-8bdc-44e6-8fad-13ed6748d685)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685"
Jul 17 05:23:03.964292 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:03.964243    2582 scope.go:117] "RemoveContainer" containerID="0aee60fc705c361cc869c4f622eb09c188324407626715e8d8c4cdd7f0148827"
Jul 17 05:23:03.964759 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:03.964495    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ks99l_odh-ai-gateway-infra(5a38b22e-8bdc-44e6-8fad-13ed6748d685)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685"
Jul 17 05:23:03.965789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:03.965765    2582 generic.go:356] "Generic (PLEG): container finished" podID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerID="d31856fd2e5cc836e875d5adf058431a996ffc7e259a31278f229e1074fc1291" exitCode=6
Jul 17 05:23:03.965891 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:03.965808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" event={"ID":"8dd8d635-d629-4eb9-b54f-7d0e1f423b06","Type":"ContainerDied","Data":"d31856fd2e5cc836e875d5adf058431a996ffc7e259a31278f229e1074fc1291"}
Jul 17 05:23:03.965891 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:03.965852    2582 scope.go:117] "RemoveContainer" containerID="93f9347998aaec49315a28a872a82b9b9d9bc8e7c3351ef04157733071ae873e"
Jul 17 05:23:03.966118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:03.966103    2582 scope.go:117] "RemoveContainer" containerID="d31856fd2e5cc836e875d5adf058431a996ffc7e259a31278f229e1074fc1291"
Jul 17 05:23:03.966318 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:03.966301    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z_odh-ai-gateway-infra(8dd8d635-d629-4eb9-b54f-7d0e1f423b06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06"
Jul 17 05:23:05.003407 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.003369    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"]
Jul 17 05:23:05.139607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.139576    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:23:05.172657 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.172606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8d635-d629-4eb9-b54f-7d0e1f423b06-maas-api-service-ca\") pod \"8dd8d635-d629-4eb9-b54f-7d0e1f423b06\" (UID: \"8dd8d635-d629-4eb9-b54f-7d0e1f423b06\") "
Jul 17 05:23:05.172964 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.172945    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd8d635-d629-4eb9-b54f-7d0e1f423b06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8dd8d635-d629-4eb9-b54f-7d0e1f423b06" (UID: "8dd8d635-d629-4eb9-b54f-7d0e1f423b06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:23:05.274162 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.274067    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8d635-d629-4eb9-b54f-7d0e1f423b06-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:23:05.978194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.978167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"
Jul 17 05:23:05.978194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.978182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z" event={"ID":"8dd8d635-d629-4eb9-b54f-7d0e1f423b06","Type":"ContainerDied","Data":"61dfeb484d62c7fddec373cdbad0dc4297d1d81f53820def19883cedd4b80948"}
Jul 17 05:23:05.978419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.978247    2582 scope.go:117] "RemoveContainer" containerID="d31856fd2e5cc836e875d5adf058431a996ffc7e259a31278f229e1074fc1291"
Jul 17 05:23:05.998730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:05.998701    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"]
Jul 17 05:23:06.002013 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:06.001987    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-bvd6z"]
Jul 17 05:23:06.730017 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:06.729984    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" path="/var/lib/kubelet/pods/8dd8d635-d629-4eb9-b54f-7d0e1f423b06/volumes"
Jul 17 05:23:15.724733 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:15.724699    2582 scope.go:117] "RemoveContainer" containerID="0aee60fc705c361cc869c4f622eb09c188324407626715e8d8c4cdd7f0148827"
Jul 17 05:23:16.021875 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.021839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" event={"ID":"5a38b22e-8bdc-44e6-8fad-13ed6748d685","Type":"ContainerStarted","Data":"e208f975b1b25296c84e47aebf8c2a01d6ad79a9447d93c140b1ab3faf490ff9"}
Jul 17 05:23:16.036357 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.036304    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" podStartSLOduration=16.036286877 podStartE2EDuration="16.036286877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:23:16.035812895 +0000 UTC m=+1999.772875231" watchObservedRunningTime="2026-07-17 05:23:16.036286877 +0000 UTC m=+1999.773349210"
Jul 17 05:23:16.600394 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600301    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"]
Jul 17 05:23:16.600665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600651    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.600665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.600799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600689    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.600799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600695    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.600799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600747    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.600799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600757    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.600799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.600763    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:16.603756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.603734    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:16.611455 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.611152    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"]
Jul 17 05:23:16.663897 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.663849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7470852e-5e16-4dc7-a6f9-999e6bb840a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k\" (UID: \"7470852e-5e16-4dc7-a6f9-999e6bb840a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:16.765111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.765069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7470852e-5e16-4dc7-a6f9-999e6bb840a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k\" (UID: \"7470852e-5e16-4dc7-a6f9-999e6bb840a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:16.765879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.765853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7470852e-5e16-4dc7-a6f9-999e6bb840a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k\" (UID: \"7470852e-5e16-4dc7-a6f9-999e6bb840a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:16.916552 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:16.916458    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:17.029853 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:17.029818    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerID="e208f975b1b25296c84e47aebf8c2a01d6ad79a9447d93c140b1ab3faf490ff9" exitCode=6
Jul 17 05:23:17.030043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:17.029886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" event={"ID":"5a38b22e-8bdc-44e6-8fad-13ed6748d685","Type":"ContainerDied","Data":"e208f975b1b25296c84e47aebf8c2a01d6ad79a9447d93c140b1ab3faf490ff9"}
Jul 17 05:23:17.030043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:17.029951    2582 scope.go:117] "RemoveContainer" containerID="0aee60fc705c361cc869c4f622eb09c188324407626715e8d8c4cdd7f0148827"
Jul 17 05:23:17.030289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:17.030251    2582 scope.go:117] "RemoveContainer" containerID="e208f975b1b25296c84e47aebf8c2a01d6ad79a9447d93c140b1ab3faf490ff9"
Jul 17 05:23:17.030543 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:17.030518    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-ks99l_odh-ai-gateway-infra(5a38b22e-8bdc-44e6-8fad-13ed6748d685)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685"
Jul 17 05:23:17.047020 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:17.046991    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"]
Jul 17 05:23:17.058844 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:23:17.058802    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7470852e_5e16_4dc7_a6f9_999e6bb840a9.slice/crio-e441c60fcc1f8707d3b4ed86f90509ea4caa6e9fae20e500393e3343d972750a WatchSource:0}: Error finding container e441c60fcc1f8707d3b4ed86f90509ea4caa6e9fae20e500393e3343d972750a: Status 404 returned error can't find the container with id e441c60fcc1f8707d3b4ed86f90509ea4caa6e9fae20e500393e3343d972750a
Jul 17 05:23:17.067818 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:17.067792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"]
Jul 17 05:23:18.035941 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.035903    2582 generic.go:356] "Generic (PLEG): container finished" podID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerID="f7225c2c4f7e7dbc383de1ea5cbd49512d96017400cf5570c88ad4b8679d8a4a" exitCode=7
Jul 17 05:23:18.036386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.035992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" event={"ID":"7470852e-5e16-4dc7-a6f9-999e6bb840a9","Type":"ContainerDied","Data":"f7225c2c4f7e7dbc383de1ea5cbd49512d96017400cf5570c88ad4b8679d8a4a"}
Jul 17 05:23:18.036386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.036051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" event={"ID":"7470852e-5e16-4dc7-a6f9-999e6bb840a9","Type":"ContainerStarted","Data":"e441c60fcc1f8707d3b4ed86f90509ea4caa6e9fae20e500393e3343d972750a"}
Jul 17 05:23:18.036386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.036183    2582 scope.go:117] "RemoveContainer" containerID="f7225c2c4f7e7dbc383de1ea5cbd49512d96017400cf5570c88ad4b8679d8a4a"
Jul 17 05:23:18.166950 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.166919    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:18.277015 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.276978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a38b22e-8bdc-44e6-8fad-13ed6748d685-maas-api-service-ca\") pod \"5a38b22e-8bdc-44e6-8fad-13ed6748d685\" (UID: \"5a38b22e-8bdc-44e6-8fad-13ed6748d685\") "
Jul 17 05:23:18.277422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.277401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a38b22e-8bdc-44e6-8fad-13ed6748d685-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a38b22e-8bdc-44e6-8fad-13ed6748d685" (UID: "5a38b22e-8bdc-44e6-8fad-13ed6748d685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:23:18.378572 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:18.378532    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a38b22e-8bdc-44e6-8fad-13ed6748d685-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:23:19.043520 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.043483    2582 generic.go:356] "Generic (PLEG): container finished" podID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerID="818baaf859c5a10a582c4517fe289e67eac80391ed943ef748abc50868578e7a" exitCode=7
Jul 17 05:23:19.043987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.043572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" event={"ID":"7470852e-5e16-4dc7-a6f9-999e6bb840a9","Type":"ContainerDied","Data":"818baaf859c5a10a582c4517fe289e67eac80391ed943ef748abc50868578e7a"}
Jul 17 05:23:19.043987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.043646    2582 scope.go:117] "RemoveContainer" containerID="f7225c2c4f7e7dbc383de1ea5cbd49512d96017400cf5570c88ad4b8679d8a4a"
Jul 17 05:23:19.043987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.043797    2582 scope.go:117] "RemoveContainer" containerID="818baaf859c5a10a582c4517fe289e67eac80391ed943ef748abc50868578e7a"
Jul 17 05:23:19.044166 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:19.044004    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k_odh-ai-gateway-infra(7470852e-5e16-4dc7-a6f9-999e6bb840a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9"
Jul 17 05:23:19.044831 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.044806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l" event={"ID":"5a38b22e-8bdc-44e6-8fad-13ed6748d685","Type":"ContainerDied","Data":"e95abda1ff0c9ffbd6e5a596c429c798815ee2205f97aa388566eca7e8c2a390"}
Jul 17 05:23:19.044879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.044830    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"
Jul 17 05:23:19.054462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.054439    2582 scope.go:117] "RemoveContainer" containerID="e208f975b1b25296c84e47aebf8c2a01d6ad79a9447d93c140b1ab3faf490ff9"
Jul 17 05:23:19.075359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.075323    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"]
Jul 17 05:23:19.080425 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:19.080399    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-ks99l"]
Jul 17 05:23:20.050943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:20.050910    2582 scope.go:117] "RemoveContainer" containerID="818baaf859c5a10a582c4517fe289e67eac80391ed943ef748abc50868578e7a"
Jul 17 05:23:20.051409 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:20.051143    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k_odh-ai-gateway-infra(7470852e-5e16-4dc7-a6f9-999e6bb840a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9"
Jul 17 05:23:20.730770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:20.730734    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" path="/var/lib/kubelet/pods/5a38b22e-8bdc-44e6-8fad-13ed6748d685/volumes"
Jul 17 05:23:34.724920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:34.724881    2582 scope.go:117] "RemoveContainer" containerID="818baaf859c5a10a582c4517fe289e67eac80391ed943ef748abc50868578e7a"
Jul 17 05:23:35.115692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:35.115594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" event={"ID":"7470852e-5e16-4dc7-a6f9-999e6bb840a9","Type":"ContainerStarted","Data":"0ec8b96317f7bbfdb1ab6ce7196e7b61bec902c362ec82f8b3af7b473745e12d"}
Jul 17 05:23:35.131454 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:35.131403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" podStartSLOduration=19.131386724 podStartE2EDuration="19.131386724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:23:35.130138478 +0000 UTC m=+2018.867200838" watchObservedRunningTime="2026-07-17 05:23:35.131386724 +0000 UTC m=+2018.868449060"
Jul 17 05:23:36.121799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:36.121759    2582 generic.go:356] "Generic (PLEG): container finished" podID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerID="0ec8b96317f7bbfdb1ab6ce7196e7b61bec902c362ec82f8b3af7b473745e12d" exitCode=7
Jul 17 05:23:36.122306 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:36.121817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" event={"ID":"7470852e-5e16-4dc7-a6f9-999e6bb840a9","Type":"ContainerDied","Data":"0ec8b96317f7bbfdb1ab6ce7196e7b61bec902c362ec82f8b3af7b473745e12d"}
Jul 17 05:23:36.122306 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:36.121880    2582 scope.go:117] "RemoveContainer" containerID="818baaf859c5a10a582c4517fe289e67eac80391ed943ef748abc50868578e7a"
Jul 17 05:23:36.122306 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:36.122153    2582 scope.go:117] "RemoveContainer" containerID="0ec8b96317f7bbfdb1ab6ce7196e7b61bec902c362ec82f8b3af7b473745e12d"
Jul 17 05:23:36.122453 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:36.122380    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k_odh-ai-gateway-infra(7470852e-5e16-4dc7-a6f9-999e6bb840a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9"
Jul 17 05:23:36.147464 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:36.147428    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"]
Jul 17 05:23:37.273611 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:37.273584    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:37.437033 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:37.436932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7470852e-5e16-4dc7-a6f9-999e6bb840a9-maas-api-service-ca\") pod \"7470852e-5e16-4dc7-a6f9-999e6bb840a9\" (UID: \"7470852e-5e16-4dc7-a6f9-999e6bb840a9\") "
Jul 17 05:23:37.437406 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:37.437381    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7470852e-5e16-4dc7-a6f9-999e6bb840a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7470852e-5e16-4dc7-a6f9-999e6bb840a9" (UID: "7470852e-5e16-4dc7-a6f9-999e6bb840a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:23:37.537912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:37.537873    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7470852e-5e16-4dc7-a6f9-999e6bb840a9-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:23:38.137746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.137715    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"
Jul 17 05:23:38.137746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.137739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k" event={"ID":"7470852e-5e16-4dc7-a6f9-999e6bb840a9","Type":"ContainerDied","Data":"e441c60fcc1f8707d3b4ed86f90509ea4caa6e9fae20e500393e3343d972750a"}
Jul 17 05:23:38.138038 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.137828    2582 scope.go:117] "RemoveContainer" containerID="0ec8b96317f7bbfdb1ab6ce7196e7b61bec902c362ec82f8b3af7b473745e12d"
Jul 17 05:23:38.163633 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.163595    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"]
Jul 17 05:23:38.169283 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.169241    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-vxq5k"]
Jul 17 05:23:38.526363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526324    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"]
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526605    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526616    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526624    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526630    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526639    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526645    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526655    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526660    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd8d635-d629-4eb9-b54f-7d0e1f423b06" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526669    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526674    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526679    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526684    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526735    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526742    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526749    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526754    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526759    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a38b22e-8bdc-44e6-8fad-13ed6748d685" containerName="revoke-keys"
Jul 17 05:23:38.526767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.526764    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:38.530930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.530913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:23:38.533476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.533453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:23:38.533612 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.533477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:23:38.536988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.536959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"]
Jul 17 05:23:38.646414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.646372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af3b99c-39d5-471a-9d0f-c98d889e1a85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7\" (UID: \"2af3b99c-39d5-471a-9d0f-c98d889e1a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:23:38.729553 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.729522    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" path="/var/lib/kubelet/pods/7470852e-5e16-4dc7-a6f9-999e6bb840a9/volumes"
Jul 17 05:23:38.747796 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.747759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af3b99c-39d5-471a-9d0f-c98d889e1a85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7\" (UID: \"2af3b99c-39d5-471a-9d0f-c98d889e1a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:23:38.748400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.748380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af3b99c-39d5-471a-9d0f-c98d889e1a85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7\" (UID: \"2af3b99c-39d5-471a-9d0f-c98d889e1a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:23:38.843406 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.843309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:23:38.978921 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:38.978877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"]
Jul 17 05:23:38.982904 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:23:38.982876    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af3b99c_39d5_471a_9d0f_c98d889e1a85.slice/crio-26c865b815a263d44a980b9860d40c844951df3e77c7965b604ab7deb04d469c WatchSource:0}: Error finding container 26c865b815a263d44a980b9860d40c844951df3e77c7965b604ab7deb04d469c: Status 404 returned error can't find the container with id 26c865b815a263d44a980b9860d40c844951df3e77c7965b604ab7deb04d469c
Jul 17 05:23:39.143846 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:39.143756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" event={"ID":"2af3b99c-39d5-471a-9d0f-c98d889e1a85","Type":"ContainerStarted","Data":"26c865b815a263d44a980b9860d40c844951df3e77c7965b604ab7deb04d469c"}
Jul 17 05:23:40.149508 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:40.149469    2582 generic.go:356] "Generic (PLEG): container finished" podID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerID="5bd5c3b6ce21448bbf2bcae6039319822fbb921babddcac9a35cd4978883f5dd" exitCode=6
Jul 17 05:23:40.149958 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:40.149534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" event={"ID":"2af3b99c-39d5-471a-9d0f-c98d889e1a85","Type":"ContainerDied","Data":"5bd5c3b6ce21448bbf2bcae6039319822fbb921babddcac9a35cd4978883f5dd"}
Jul 17 05:23:40.149958 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:40.149733    2582 scope.go:117] "RemoveContainer" containerID="5bd5c3b6ce21448bbf2bcae6039319822fbb921babddcac9a35cd4978883f5dd"
Jul 17 05:23:41.155960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:41.155922    2582 generic.go:356] "Generic (PLEG): container finished" podID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerID="baf5f5913c27f809685f7dd3930d68e1742b1b212d7bf271264846f59dbb7cd2" exitCode=6
Jul 17 05:23:41.156445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:41.156015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" event={"ID":"2af3b99c-39d5-471a-9d0f-c98d889e1a85","Type":"ContainerDied","Data":"baf5f5913c27f809685f7dd3930d68e1742b1b212d7bf271264846f59dbb7cd2"}
Jul 17 05:23:41.156445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:41.156078    2582 scope.go:117] "RemoveContainer" containerID="5bd5c3b6ce21448bbf2bcae6039319822fbb921babddcac9a35cd4978883f5dd"
Jul 17 05:23:41.156445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:41.156177    2582 scope.go:117] "RemoveContainer" containerID="baf5f5913c27f809685f7dd3930d68e1742b1b212d7bf271264846f59dbb7cd2"
Jul 17 05:23:41.156573 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:41.156441    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7_odh-ai-gateway-infra(2af3b99c-39d5-471a-9d0f-c98d889e1a85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85"
Jul 17 05:23:42.162308 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:42.162250    2582 scope.go:117] "RemoveContainer" containerID="baf5f5913c27f809685f7dd3930d68e1742b1b212d7bf271264846f59dbb7cd2"
Jul 17 05:23:42.162693 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:42.162493    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7_odh-ai-gateway-infra(2af3b99c-39d5-471a-9d0f-c98d889e1a85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85"
Jul 17 05:23:50.379132 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.379096    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"]
Jul 17 05:23:50.379674 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.379418    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:50.379674 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.379430    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7470852e-5e16-4dc7-a6f9-999e6bb840a9" containerName="revoke-keys"
Jul 17 05:23:50.383809 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.383791    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:23:50.388899 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.388803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"]
Jul 17 05:23:50.444646 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.444596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518ba99c-bdc8-4a8b-ae23-43b686211098-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-lhx98\" (UID: \"518ba99c-bdc8-4a8b-ae23-43b686211098\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:23:50.545635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.545597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518ba99c-bdc8-4a8b-ae23-43b686211098-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-lhx98\" (UID: \"518ba99c-bdc8-4a8b-ae23-43b686211098\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:23:50.546237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.546213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518ba99c-bdc8-4a8b-ae23-43b686211098-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-lhx98\" (UID: \"518ba99c-bdc8-4a8b-ae23-43b686211098\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:23:50.695061 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.695028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:23:50.827768 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:50.827742    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"]
Jul 17 05:23:50.830393 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:23:50.830360    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518ba99c_bdc8_4a8b_ae23_43b686211098.slice/crio-1120c1fc545c9467b281b0a1284831b68ca916ecf8f3b29b51b506779dcc9e01 WatchSource:0}: Error finding container 1120c1fc545c9467b281b0a1284831b68ca916ecf8f3b29b51b506779dcc9e01: Status 404 returned error can't find the container with id 1120c1fc545c9467b281b0a1284831b68ca916ecf8f3b29b51b506779dcc9e01
Jul 17 05:23:51.201663 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:51.201628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerStarted","Data":"1174402a2e95a7ab5e331caf83661583dc888e16d00c05f79dbbb26f1e18c5ec"}
Jul 17 05:23:51.201836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:51.201690    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerStarted","Data":"1120c1fc545c9467b281b0a1284831b68ca916ecf8f3b29b51b506779dcc9e01"}
Jul 17 05:23:51.216559 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:51.216505    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" podStartSLOduration=1.216488765 podStartE2EDuration="1.216488765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:23:51.215451176 +0000 UTC m=+2034.952513535" watchObservedRunningTime="2026-07-17 05:23:51.216488765 +0000 UTC m=+2034.953551101"
Jul 17 05:23:52.207839 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:52.207798    2582 generic.go:356] "Generic (PLEG): container finished" podID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerID="1174402a2e95a7ab5e331caf83661583dc888e16d00c05f79dbbb26f1e18c5ec" exitCode=6
Jul 17 05:23:52.208285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:52.207866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerDied","Data":"1174402a2e95a7ab5e331caf83661583dc888e16d00c05f79dbbb26f1e18c5ec"}
Jul 17 05:23:52.208285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:52.208140    2582 scope.go:117] "RemoveContainer" containerID="1174402a2e95a7ab5e331caf83661583dc888e16d00c05f79dbbb26f1e18c5ec"
Jul 17 05:23:53.213984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:53.213944    2582 generic.go:356] "Generic (PLEG): container finished" podID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerID="3671ee1d84f9f08d5189fdd9d454a990047915ba2c73f969d2623c213c91c322" exitCode=6
Jul 17 05:23:53.213984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:53.213987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerDied","Data":"3671ee1d84f9f08d5189fdd9d454a990047915ba2c73f969d2623c213c91c322"}
Jul 17 05:23:53.214558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:53.214061    2582 scope.go:117] "RemoveContainer" containerID="1174402a2e95a7ab5e331caf83661583dc888e16d00c05f79dbbb26f1e18c5ec"
Jul 17 05:23:53.214558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:53.214313    2582 scope.go:117] "RemoveContainer" containerID="3671ee1d84f9f08d5189fdd9d454a990047915ba2c73f969d2623c213c91c322"
Jul 17 05:23:53.214653 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:53.214570    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-lhx98_odh-ai-gateway-infra(518ba99c-bdc8-4a8b-ae23-43b686211098)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098"
Jul 17 05:23:54.221355 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:54.221322    2582 scope.go:117] "RemoveContainer" containerID="3671ee1d84f9f08d5189fdd9d454a990047915ba2c73f969d2623c213c91c322"
Jul 17 05:23:54.221832 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:54.221550    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-lhx98_odh-ai-gateway-infra(518ba99c-bdc8-4a8b-ae23-43b686211098)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098"
Jul 17 05:23:57.724570 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:57.724536    2582 scope.go:117] "RemoveContainer" containerID="baf5f5913c27f809685f7dd3930d68e1742b1b212d7bf271264846f59dbb7cd2"
Jul 17 05:23:58.243055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:58.243021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" event={"ID":"2af3b99c-39d5-471a-9d0f-c98d889e1a85","Type":"ContainerStarted","Data":"d8ad33b8dcf868fc2f2d9dd673d9afec1355b75b28fab302e711796336d09506"}
Jul 17 05:23:58.260319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:58.260239    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" podStartSLOduration=20.260220805 podStartE2EDuration="20.260220805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:23:58.258852071 +0000 UTC m=+2041.995914426" watchObservedRunningTime="2026-07-17 05:23:58.260220805 +0000 UTC m=+2041.997283142"
Jul 17 05:23:59.248990 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:59.248959    2582 generic.go:356] "Generic (PLEG): container finished" podID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerID="d8ad33b8dcf868fc2f2d9dd673d9afec1355b75b28fab302e711796336d09506" exitCode=6
Jul 17 05:23:59.249440 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:59.249031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" event={"ID":"2af3b99c-39d5-471a-9d0f-c98d889e1a85","Type":"ContainerDied","Data":"d8ad33b8dcf868fc2f2d9dd673d9afec1355b75b28fab302e711796336d09506"}
Jul 17 05:23:59.249440 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:59.249095    2582 scope.go:117] "RemoveContainer" containerID="baf5f5913c27f809685f7dd3930d68e1742b1b212d7bf271264846f59dbb7cd2"
Jul 17 05:23:59.249440 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:59.249396    2582 scope.go:117] "RemoveContainer" containerID="d8ad33b8dcf868fc2f2d9dd673d9afec1355b75b28fab302e711796336d09506"
Jul 17 05:23:59.249643 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:23:59.249625    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7_odh-ai-gateway-infra(2af3b99c-39d5-471a-9d0f-c98d889e1a85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85"
Jul 17 05:23:59.274233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:23:59.274199    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"]
Jul 17 05:24:00.389004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:00.388980    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:24:00.422638 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:00.422602    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af3b99c-39d5-471a-9d0f-c98d889e1a85-maas-api-service-ca\") pod \"2af3b99c-39d5-471a-9d0f-c98d889e1a85\" (UID: \"2af3b99c-39d5-471a-9d0f-c98d889e1a85\") "
Jul 17 05:24:00.422973 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:00.422949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af3b99c-39d5-471a-9d0f-c98d889e1a85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2af3b99c-39d5-471a-9d0f-c98d889e1a85" (UID: "2af3b99c-39d5-471a-9d0f-c98d889e1a85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:24:00.523145 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:00.523056    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af3b99c-39d5-471a-9d0f-c98d889e1a85-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:24:01.260670 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:01.260627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7" event={"ID":"2af3b99c-39d5-471a-9d0f-c98d889e1a85","Type":"ContainerDied","Data":"26c865b815a263d44a980b9860d40c844951df3e77c7965b604ab7deb04d469c"}
Jul 17 05:24:01.260670 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:01.260652    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"
Jul 17 05:24:01.260918 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:01.260705    2582 scope.go:117] "RemoveContainer" containerID="d8ad33b8dcf868fc2f2d9dd673d9afec1355b75b28fab302e711796336d09506"
Jul 17 05:24:01.277830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:01.277799    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"]
Jul 17 05:24:01.284934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:01.284906    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-6p2f7"]
Jul 17 05:24:02.730432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:02.730398    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" path="/var/lib/kubelet/pods/2af3b99c-39d5-471a-9d0f-c98d889e1a85/volumes"
Jul 17 05:24:08.724778 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:08.724748    2582 scope.go:117] "RemoveContainer" containerID="3671ee1d84f9f08d5189fdd9d454a990047915ba2c73f969d2623c213c91c322"
Jul 17 05:24:09.295770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:09.295671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerStarted","Data":"89d175b66d1fa6d90946db1976b18968a6eb48eba3b8a4ca9562a3cd30c4698d"}
Jul 17 05:24:10.301942 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:10.301905    2582 generic.go:356] "Generic (PLEG): container finished" podID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerID="89d175b66d1fa6d90946db1976b18968a6eb48eba3b8a4ca9562a3cd30c4698d" exitCode=6
Jul 17 05:24:10.302409 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:10.301976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerDied","Data":"89d175b66d1fa6d90946db1976b18968a6eb48eba3b8a4ca9562a3cd30c4698d"}
Jul 17 05:24:10.302409 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:10.302061    2582 scope.go:117] "RemoveContainer" containerID="3671ee1d84f9f08d5189fdd9d454a990047915ba2c73f969d2623c213c91c322"
Jul 17 05:24:10.302409 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:10.302379    2582 scope.go:117] "RemoveContainer" containerID="89d175b66d1fa6d90946db1976b18968a6eb48eba3b8a4ca9562a3cd30c4698d"
Jul 17 05:24:10.302614 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:10.302595    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-lhx98_odh-ai-gateway-infra(518ba99c-bdc8-4a8b-ae23-43b686211098)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098"
Jul 17 05:24:10.327955 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:10.327912    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"]
Jul 17 05:24:11.439933 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:11.439907    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:24:11.500844 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:11.500807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518ba99c-bdc8-4a8b-ae23-43b686211098-maas-api-service-ca\") pod \"518ba99c-bdc8-4a8b-ae23-43b686211098\" (UID: \"518ba99c-bdc8-4a8b-ae23-43b686211098\") "
Jul 17 05:24:11.501221 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:11.501194    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518ba99c-bdc8-4a8b-ae23-43b686211098-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "518ba99c-bdc8-4a8b-ae23-43b686211098" (UID: "518ba99c-bdc8-4a8b-ae23-43b686211098"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:24:11.601935 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:11.601850    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518ba99c-bdc8-4a8b-ae23-43b686211098-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:24:12.312789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:12.312758    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"
Jul 17 05:24:12.312789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:12.312781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98" event={"ID":"518ba99c-bdc8-4a8b-ae23-43b686211098","Type":"ContainerDied","Data":"1120c1fc545c9467b281b0a1284831b68ca916ecf8f3b29b51b506779dcc9e01"}
Jul 17 05:24:12.313016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:12.312846    2582 scope.go:117] "RemoveContainer" containerID="89d175b66d1fa6d90946db1976b18968a6eb48eba3b8a4ca9562a3cd30c4698d"
Jul 17 05:24:12.335600 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:12.335568    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"]
Jul 17 05:24:12.341687 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:12.341662    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lhx98"]
Jul 17 05:24:12.730858 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:12.730828    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" path="/var/lib/kubelet/pods/518ba99c-bdc8-4a8b-ae23-43b686211098/volumes"
Jul 17 05:24:16.645392 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645355    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"]
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645637    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645647    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645665    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645671    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645685    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645690    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645695    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645700    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645707    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645712    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645756    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645764    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645771    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:16.645815 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.645777    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:16.649928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.649910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:16.652427 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.652405    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:24:16.652427 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.652421    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:24:16.655452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.655427    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"]
Jul 17 05:24:16.742993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.742955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38834250-fb45-4667-a7d3-35199de49eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr\" (UID: \"38834250-fb45-4667-a7d3-35199de49eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:16.844369 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.844333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38834250-fb45-4667-a7d3-35199de49eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr\" (UID: \"38834250-fb45-4667-a7d3-35199de49eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:16.844988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.844970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38834250-fb45-4667-a7d3-35199de49eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr\" (UID: \"38834250-fb45-4667-a7d3-35199de49eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:16.961626 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:16.961588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:17.097016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:17.096983    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"]
Jul 17 05:24:17.100547 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:24:17.100519    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38834250_fb45_4667_a7d3_35199de49eee.slice/crio-86064c1742fa389038117d215cdf680dd6301d0ac69e96ed1a774ea1f6a0e4d1 WatchSource:0}: Error finding container 86064c1742fa389038117d215cdf680dd6301d0ac69e96ed1a774ea1f6a0e4d1: Status 404 returned error can't find the container with id 86064c1742fa389038117d215cdf680dd6301d0ac69e96ed1a774ea1f6a0e4d1
Jul 17 05:24:17.335901 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:17.335865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" event={"ID":"38834250-fb45-4667-a7d3-35199de49eee","Type":"ContainerStarted","Data":"9ff50211c933c9aa28124c77ad447a5189377a85f09520c0d437389108175c2a"}
Jul 17 05:24:17.336079 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:17.335922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" event={"ID":"38834250-fb45-4667-a7d3-35199de49eee","Type":"ContainerStarted","Data":"86064c1742fa389038117d215cdf680dd6301d0ac69e96ed1a774ea1f6a0e4d1"}
Jul 17 05:24:17.352181 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:17.352130    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" podStartSLOduration=1.3521125299999999 podStartE2EDuration="1.35211253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:24:17.351168231 +0000 UTC m=+2061.088230568" watchObservedRunningTime="2026-07-17 05:24:17.35211253 +0000 UTC m=+2061.089174866"
Jul 17 05:24:18.341503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:18.341463    2582 generic.go:356] "Generic (PLEG): container finished" podID="38834250-fb45-4667-a7d3-35199de49eee" containerID="9ff50211c933c9aa28124c77ad447a5189377a85f09520c0d437389108175c2a" exitCode=7
Jul 17 05:24:18.341990 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:18.341519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" event={"ID":"38834250-fb45-4667-a7d3-35199de49eee","Type":"ContainerDied","Data":"9ff50211c933c9aa28124c77ad447a5189377a85f09520c0d437389108175c2a"}
Jul 17 05:24:18.341990 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:18.341759    2582 scope.go:117] "RemoveContainer" containerID="9ff50211c933c9aa28124c77ad447a5189377a85f09520c0d437389108175c2a"
Jul 17 05:24:19.347235 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:19.347201    2582 generic.go:356] "Generic (PLEG): container finished" podID="38834250-fb45-4667-a7d3-35199de49eee" containerID="7a90fb30aa8351079e2d673c5973c6920d99867c2ca264fdd249107dd1e75b66" exitCode=7
Jul 17 05:24:19.347725 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:19.347291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" event={"ID":"38834250-fb45-4667-a7d3-35199de49eee","Type":"ContainerDied","Data":"7a90fb30aa8351079e2d673c5973c6920d99867c2ca264fdd249107dd1e75b66"}
Jul 17 05:24:19.347725 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:19.347366    2582 scope.go:117] "RemoveContainer" containerID="9ff50211c933c9aa28124c77ad447a5189377a85f09520c0d437389108175c2a"
Jul 17 05:24:19.347725 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:19.347599    2582 scope.go:117] "RemoveContainer" containerID="7a90fb30aa8351079e2d673c5973c6920d99867c2ca264fdd249107dd1e75b66"
Jul 17 05:24:19.347889 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:19.347808    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr_odh-ai-gateway-infra(38834250-fb45-4667-a7d3-35199de49eee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" podUID="38834250-fb45-4667-a7d3-35199de49eee"
Jul 17 05:24:20.353107 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:20.353077    2582 scope.go:117] "RemoveContainer" containerID="7a90fb30aa8351079e2d673c5973c6920d99867c2ca264fdd249107dd1e75b66"
Jul 17 05:24:20.353592 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:20.353242    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr_odh-ai-gateway-infra(38834250-fb45-4667-a7d3-35199de49eee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" podUID="38834250-fb45-4667-a7d3-35199de49eee"
Jul 17 05:24:34.729908 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:34.729876    2582 scope.go:117] "RemoveContainer" containerID="7a90fb30aa8351079e2d673c5973c6920d99867c2ca264fdd249107dd1e75b66"
Jul 17 05:24:35.416158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:35.416128    2582 generic.go:356] "Generic (PLEG): container finished" podID="38834250-fb45-4667-a7d3-35199de49eee" containerID="db487f2fb25a560becaaae39e21cf30501677270b3432c73b88dd03f1a54306c" exitCode=7
Jul 17 05:24:35.416364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:35.416199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" event={"ID":"38834250-fb45-4667-a7d3-35199de49eee","Type":"ContainerDied","Data":"db487f2fb25a560becaaae39e21cf30501677270b3432c73b88dd03f1a54306c"}
Jul 17 05:24:35.416364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:35.416281    2582 scope.go:117] "RemoveContainer" containerID="7a90fb30aa8351079e2d673c5973c6920d99867c2ca264fdd249107dd1e75b66"
Jul 17 05:24:35.416578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:35.416562    2582 scope.go:117] "RemoveContainer" containerID="db487f2fb25a560becaaae39e21cf30501677270b3432c73b88dd03f1a54306c"
Jul 17 05:24:35.416797 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:35.416773    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr_odh-ai-gateway-infra(38834250-fb45-4667-a7d3-35199de49eee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" podUID="38834250-fb45-4667-a7d3-35199de49eee"
Jul 17 05:24:36.442901 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:36.442859    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"]
Jul 17 05:24:36.574473 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:36.574447    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:36.700212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:36.700121    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38834250-fb45-4667-a7d3-35199de49eee-maas-api-service-ca\") pod \"38834250-fb45-4667-a7d3-35199de49eee\" (UID: \"38834250-fb45-4667-a7d3-35199de49eee\") "
Jul 17 05:24:36.700505 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:36.700480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38834250-fb45-4667-a7d3-35199de49eee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38834250-fb45-4667-a7d3-35199de49eee" (UID: "38834250-fb45-4667-a7d3-35199de49eee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:24:36.800698 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:36.800661    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38834250-fb45-4667-a7d3-35199de49eee-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:24:37.426967 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:37.426932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr" event={"ID":"38834250-fb45-4667-a7d3-35199de49eee","Type":"ContainerDied","Data":"86064c1742fa389038117d215cdf680dd6301d0ac69e96ed1a774ea1f6a0e4d1"}
Jul 17 05:24:37.426967 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:37.426950    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"
Jul 17 05:24:37.427236 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:37.427003    2582 scope.go:117] "RemoveContainer" containerID="db487f2fb25a560becaaae39e21cf30501677270b3432c73b88dd03f1a54306c"
Jul 17 05:24:37.443410 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:37.443380    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"]
Jul 17 05:24:37.452685 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:37.452656    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-6s7vr"]
Jul 17 05:24:38.578299 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578247    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"]
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578557    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578567    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578581    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578588    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578603    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578608    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578653    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578661    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578668    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af3b99c-39d5-471a-9d0f-c98d889e1a85" containerName="revoke-keys"
Jul 17 05:24:38.578684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.578676    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="518ba99c-bdc8-4a8b-ae23-43b686211098" containerName="revoke-keys"
Jul 17 05:24:38.582893 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.582873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:24:38.585436 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.585415    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:24:38.585585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.585415    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:24:38.588930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.588896    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"]
Jul 17 05:24:38.616483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.616448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e77540e-e842-4538-84b1-e58436e9a56b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22\" (UID: \"8e77540e-e842-4538-84b1-e58436e9a56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:24:38.717428 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.717386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e77540e-e842-4538-84b1-e58436e9a56b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22\" (UID: \"8e77540e-e842-4538-84b1-e58436e9a56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:24:38.718018 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.717996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e77540e-e842-4538-84b1-e58436e9a56b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22\" (UID: \"8e77540e-e842-4538-84b1-e58436e9a56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:24:38.734814 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.730399    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38834250-fb45-4667-a7d3-35199de49eee" path="/var/lib/kubelet/pods/38834250-fb45-4667-a7d3-35199de49eee/volumes"
Jul 17 05:24:38.894972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:38.894879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:24:39.025081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:39.025056    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"]
Jul 17 05:24:39.027585 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:24:39.027548    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e77540e_e842_4538_84b1_e58436e9a56b.slice/crio-2029eb10c14d0636c86664c62231597ec625c12c77c97d9fc742c15f047ad3b0 WatchSource:0}: Error finding container 2029eb10c14d0636c86664c62231597ec625c12c77c97d9fc742c15f047ad3b0: Status 404 returned error can't find the container with id 2029eb10c14d0636c86664c62231597ec625c12c77c97d9fc742c15f047ad3b0
Jul 17 05:24:39.438519 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:39.438487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" event={"ID":"8e77540e-e842-4538-84b1-e58436e9a56b","Type":"ContainerStarted","Data":"18d2f07440648f4b95b63e8a3a7f81cb98ecf0446c637e55b10411f1158ffa1f"}
Jul 17 05:24:39.438696 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:39.438535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" event={"ID":"8e77540e-e842-4538-84b1-e58436e9a56b","Type":"ContainerStarted","Data":"2029eb10c14d0636c86664c62231597ec625c12c77c97d9fc742c15f047ad3b0"}
Jul 17 05:24:39.454783 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:39.454687    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" podStartSLOduration=1.45466445 podStartE2EDuration="1.45466445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:24:39.453068435 +0000 UTC m=+2083.190130787" watchObservedRunningTime="2026-07-17 05:24:39.45466445 +0000 UTC m=+2083.191726787"
Jul 17 05:24:40.444557 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:40.444523    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e77540e-e842-4538-84b1-e58436e9a56b" containerID="18d2f07440648f4b95b63e8a3a7f81cb98ecf0446c637e55b10411f1158ffa1f" exitCode=6
Jul 17 05:24:40.444968 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:40.444618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" event={"ID":"8e77540e-e842-4538-84b1-e58436e9a56b","Type":"ContainerDied","Data":"18d2f07440648f4b95b63e8a3a7f81cb98ecf0446c637e55b10411f1158ffa1f"}
Jul 17 05:24:40.444968 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:40.444803    2582 scope.go:117] "RemoveContainer" containerID="18d2f07440648f4b95b63e8a3a7f81cb98ecf0446c637e55b10411f1158ffa1f"
Jul 17 05:24:41.450177 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:41.450146    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e77540e-e842-4538-84b1-e58436e9a56b" containerID="4c48ec5368b578598ec14aed5e1e201c4b79197315bda81b4ad616400ebaf25e" exitCode=6
Jul 17 05:24:41.450635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:41.450234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" event={"ID":"8e77540e-e842-4538-84b1-e58436e9a56b","Type":"ContainerDied","Data":"4c48ec5368b578598ec14aed5e1e201c4b79197315bda81b4ad616400ebaf25e"}
Jul 17 05:24:41.450635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:41.450324    2582 scope.go:117] "RemoveContainer" containerID="18d2f07440648f4b95b63e8a3a7f81cb98ecf0446c637e55b10411f1158ffa1f"
Jul 17 05:24:41.450635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:41.450451    2582 scope.go:117] "RemoveContainer" containerID="4c48ec5368b578598ec14aed5e1e201c4b79197315bda81b4ad616400ebaf25e"
Jul 17 05:24:41.450752 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:41.450659    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22_odh-ai-gateway-infra(8e77540e-e842-4538-84b1-e58436e9a56b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" podUID="8e77540e-e842-4538-84b1-e58436e9a56b"
Jul 17 05:24:42.456556 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:42.456525    2582 scope.go:117] "RemoveContainer" containerID="4c48ec5368b578598ec14aed5e1e201c4b79197315bda81b4ad616400ebaf25e"
Jul 17 05:24:42.456924 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:42.456758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22_odh-ai-gateway-infra(8e77540e-e842-4538-84b1-e58436e9a56b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" podUID="8e77540e-e842-4538-84b1-e58436e9a56b"
Jul 17 05:24:50.426852 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.426811    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"]
Jul 17 05:24:50.427300 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.427125    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:50.427300 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.427136    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:50.427300 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.427189    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38834250-fb45-4667-a7d3-35199de49eee" containerName="revoke-keys"
Jul 17 05:24:50.431347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.431325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:24:50.437227 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.437194    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"]
Jul 17 05:24:50.516633 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.516587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29a6bdd-7d56-4351-b353-793b88418a5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-kd82t\" (UID: \"a29a6bdd-7d56-4351-b353-793b88418a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:24:50.617063 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.617019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29a6bdd-7d56-4351-b353-793b88418a5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-kd82t\" (UID: \"a29a6bdd-7d56-4351-b353-793b88418a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:24:50.617661 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.617642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29a6bdd-7d56-4351-b353-793b88418a5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-kd82t\" (UID: \"a29a6bdd-7d56-4351-b353-793b88418a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:24:50.743765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.743725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:24:50.881368 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:50.881310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"]
Jul 17 05:24:50.883837 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:24:50.883805    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29a6bdd_7d56_4351_b353_793b88418a5d.slice/crio-342d607530db194bb90ded723af8495eab830ad9d9a2b047d1a7594601af081e WatchSource:0}: Error finding container 342d607530db194bb90ded723af8495eab830ad9d9a2b047d1a7594601af081e: Status 404 returned error can't find the container with id 342d607530db194bb90ded723af8495eab830ad9d9a2b047d1a7594601af081e
Jul 17 05:24:51.501041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:51.501003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" event={"ID":"a29a6bdd-7d56-4351-b353-793b88418a5d","Type":"ContainerStarted","Data":"538972a0b6844c08cdcf480e8d97a16509cefaecf6e5a76b6a160ffb7fcdba0b"}
Jul 17 05:24:51.501811 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:51.501776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" event={"ID":"a29a6bdd-7d56-4351-b353-793b88418a5d","Type":"ContainerStarted","Data":"342d607530db194bb90ded723af8495eab830ad9d9a2b047d1a7594601af081e"}
Jul 17 05:24:51.518422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:51.518376    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" podStartSLOduration=1.518360393 podStartE2EDuration="1.518360393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:24:51.516978247 +0000 UTC m=+2095.254040610" watchObservedRunningTime="2026-07-17 05:24:51.518360393 +0000 UTC m=+2095.255422743"
Jul 17 05:24:52.506091 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:52.506050    2582 generic.go:356] "Generic (PLEG): container finished" podID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerID="538972a0b6844c08cdcf480e8d97a16509cefaecf6e5a76b6a160ffb7fcdba0b" exitCode=6
Jul 17 05:24:52.506519 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:52.506138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" event={"ID":"a29a6bdd-7d56-4351-b353-793b88418a5d","Type":"ContainerDied","Data":"538972a0b6844c08cdcf480e8d97a16509cefaecf6e5a76b6a160ffb7fcdba0b"}
Jul 17 05:24:52.506519 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:52.506383    2582 scope.go:117] "RemoveContainer" containerID="538972a0b6844c08cdcf480e8d97a16509cefaecf6e5a76b6a160ffb7fcdba0b"
Jul 17 05:24:53.512173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:53.512132    2582 generic.go:356] "Generic (PLEG): container finished" podID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerID="eae7c7a0eb9592ee62056d34790993631984efe01ec0a2f70ef46fc6c3f017a9" exitCode=6
Jul 17 05:24:53.512610 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:53.512204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" event={"ID":"a29a6bdd-7d56-4351-b353-793b88418a5d","Type":"ContainerDied","Data":"eae7c7a0eb9592ee62056d34790993631984efe01ec0a2f70ef46fc6c3f017a9"}
Jul 17 05:24:53.512610 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:53.512316    2582 scope.go:117] "RemoveContainer" containerID="538972a0b6844c08cdcf480e8d97a16509cefaecf6e5a76b6a160ffb7fcdba0b"
Jul 17 05:24:53.512610 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:53.512541    2582 scope.go:117] "RemoveContainer" containerID="eae7c7a0eb9592ee62056d34790993631984efe01ec0a2f70ef46fc6c3f017a9"
Jul 17 05:24:53.512800 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:53.512777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-kd82t_odh-ai-gateway-infra(a29a6bdd-7d56-4351-b353-793b88418a5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d"
Jul 17 05:24:54.518098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:54.518064    2582 scope.go:117] "RemoveContainer" containerID="eae7c7a0eb9592ee62056d34790993631984efe01ec0a2f70ef46fc6c3f017a9"
Jul 17 05:24:54.518524 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:54.518238    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-kd82t_odh-ai-gateway-infra(a29a6bdd-7d56-4351-b353-793b88418a5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d"
Jul 17 05:24:56.784156 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:56.784131    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:24:56.787560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:56.787540    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:24:57.725382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:57.725352    2582 scope.go:117] "RemoveContainer" containerID="4c48ec5368b578598ec14aed5e1e201c4b79197315bda81b4ad616400ebaf25e"
Jul 17 05:24:58.535758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:58.535729    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e77540e-e842-4538-84b1-e58436e9a56b" containerID="ddbfbe3ebd28562370a4af9e754af9b10c78224aa5eeaf0d882a4e6e84c50606" exitCode=6
Jul 17 05:24:58.536186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:58.535788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" event={"ID":"8e77540e-e842-4538-84b1-e58436e9a56b","Type":"ContainerDied","Data":"ddbfbe3ebd28562370a4af9e754af9b10c78224aa5eeaf0d882a4e6e84c50606"}
Jul 17 05:24:58.536186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:58.535838    2582 scope.go:117] "RemoveContainer" containerID="4c48ec5368b578598ec14aed5e1e201c4b79197315bda81b4ad616400ebaf25e"
Jul 17 05:24:58.536186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:58.536087    2582 scope.go:117] "RemoveContainer" containerID="ddbfbe3ebd28562370a4af9e754af9b10c78224aa5eeaf0d882a4e6e84c50606"
Jul 17 05:24:58.536342 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:24:58.536315    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22_odh-ai-gateway-infra(8e77540e-e842-4538-84b1-e58436e9a56b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" podUID="8e77540e-e842-4538-84b1-e58436e9a56b"
Jul 17 05:24:59.564016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:59.563981    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"]
Jul 17 05:24:59.690751 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:59.690722    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:24:59.794939 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:59.794903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e77540e-e842-4538-84b1-e58436e9a56b-maas-api-service-ca\") pod \"8e77540e-e842-4538-84b1-e58436e9a56b\" (UID: \"8e77540e-e842-4538-84b1-e58436e9a56b\") "
Jul 17 05:24:59.795337 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:59.795313    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e77540e-e842-4538-84b1-e58436e9a56b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e77540e-e842-4538-84b1-e58436e9a56b" (UID: "8e77540e-e842-4538-84b1-e58436e9a56b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:24:59.896179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:24:59.896087    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e77540e-e842-4538-84b1-e58436e9a56b-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:25:00.547765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:00.547730    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"
Jul 17 05:25:00.547765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:00.547745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22" event={"ID":"8e77540e-e842-4538-84b1-e58436e9a56b","Type":"ContainerDied","Data":"2029eb10c14d0636c86664c62231597ec625c12c77c97d9fc742c15f047ad3b0"}
Jul 17 05:25:00.548001 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:00.547812    2582 scope.go:117] "RemoveContainer" containerID="ddbfbe3ebd28562370a4af9e754af9b10c78224aa5eeaf0d882a4e6e84c50606"
Jul 17 05:25:00.569814 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:00.569788    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"]
Jul 17 05:25:00.580200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:00.580168    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lkt22"]
Jul 17 05:25:00.729863 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:00.729823    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" path="/var/lib/kubelet/pods/8e77540e-e842-4538-84b1-e58436e9a56b/volumes"
Jul 17 05:25:06.726108 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:06.726067    2582 scope.go:117] "RemoveContainer" containerID="eae7c7a0eb9592ee62056d34790993631984efe01ec0a2f70ef46fc6c3f017a9"
Jul 17 05:25:07.581252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:07.581168    2582 generic.go:356] "Generic (PLEG): container finished" podID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerID="1112344bd8b96cde7ee8421b144ebc3edc0af1cd5f2e0e244b254f6a756d9337" exitCode=6
Jul 17 05:25:07.581435 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:07.581244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" event={"ID":"a29a6bdd-7d56-4351-b353-793b88418a5d","Type":"ContainerDied","Data":"1112344bd8b96cde7ee8421b144ebc3edc0af1cd5f2e0e244b254f6a756d9337"}
Jul 17 05:25:07.581435 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:07.581332    2582 scope.go:117] "RemoveContainer" containerID="eae7c7a0eb9592ee62056d34790993631984efe01ec0a2f70ef46fc6c3f017a9"
Jul 17 05:25:07.581671 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:07.581657    2582 scope.go:117] "RemoveContainer" containerID="1112344bd8b96cde7ee8421b144ebc3edc0af1cd5f2e0e244b254f6a756d9337"
Jul 17 05:25:07.581874 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:07.581850    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-kd82t_odh-ai-gateway-infra(a29a6bdd-7d56-4351-b353-793b88418a5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d"
Jul 17 05:25:08.605889 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:08.605853    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"]
Jul 17 05:25:08.738691 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:08.738668    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:25:08.870616 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:08.870529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29a6bdd-7d56-4351-b353-793b88418a5d-maas-api-service-ca\") pod \"a29a6bdd-7d56-4351-b353-793b88418a5d\" (UID: \"a29a6bdd-7d56-4351-b353-793b88418a5d\") "
Jul 17 05:25:08.870917 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:08.870898    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29a6bdd-7d56-4351-b353-793b88418a5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a29a6bdd-7d56-4351-b353-793b88418a5d" (UID: "a29a6bdd-7d56-4351-b353-793b88418a5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:25:08.971637 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:08.971600    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29a6bdd-7d56-4351-b353-793b88418a5d-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:25:09.591480 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:09.591450    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"
Jul 17 05:25:09.591480 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:09.591463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t" event={"ID":"a29a6bdd-7d56-4351-b353-793b88418a5d","Type":"ContainerDied","Data":"342d607530db194bb90ded723af8495eab830ad9d9a2b047d1a7594601af081e"}
Jul 17 05:25:09.591715 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:09.591549    2582 scope.go:117] "RemoveContainer" containerID="1112344bd8b96cde7ee8421b144ebc3edc0af1cd5f2e0e244b254f6a756d9337"
Jul 17 05:25:09.616866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:09.616828    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"]
Jul 17 05:25:09.623231 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:09.623197    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-kd82t"]
Jul 17 05:25:10.730107 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:10.730064    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" path="/var/lib/kubelet/pods/a29a6bdd-7d56-4351-b353-793b88418a5d/volumes"
Jul 17 05:25:16.685353 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685313    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"]
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685623    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685634    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685641    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685647    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685656    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685662    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685668    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685673    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685680    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685685    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685696    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685744    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685751    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.685756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685760    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.686209 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685766    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e77540e-e842-4538-84b1-e58436e9a56b" containerName="revoke-keys"
Jul 17 05:25:16.686209 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685772    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.686209 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.685777    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29a6bdd-7d56-4351-b353-793b88418a5d" containerName="revoke-keys"
Jul 17 05:25:16.690007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.689980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:16.692553 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.692530    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:25:16.692553 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.692553    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:25:16.696021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.695995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"]
Jul 17 05:25:16.832637 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.832584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8864b9-1069-41e5-a637-11b1b33a4c13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-thph2\" (UID: \"3b8864b9-1069-41e5-a637-11b1b33a4c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:16.933338 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.933301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8864b9-1069-41e5-a637-11b1b33a4c13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-thph2\" (UID: \"3b8864b9-1069-41e5-a637-11b1b33a4c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:16.933898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:16.933877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8864b9-1069-41e5-a637-11b1b33a4c13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-thph2\" (UID: \"3b8864b9-1069-41e5-a637-11b1b33a4c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:17.002138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:17.002098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:17.135376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:17.135344    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"]
Jul 17 05:25:17.138346 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:25:17.138312    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8864b9_1069_41e5_a637_11b1b33a4c13.slice/crio-2658451b1192b16febd9b628b5a3e344573e3c8cb8d718e38c6b5845503e1a3a WatchSource:0}: Error finding container 2658451b1192b16febd9b628b5a3e344573e3c8cb8d718e38c6b5845503e1a3a: Status 404 returned error can't find the container with id 2658451b1192b16febd9b628b5a3e344573e3c8cb8d718e38c6b5845503e1a3a
Jul 17 05:25:17.626695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:17.626597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" event={"ID":"3b8864b9-1069-41e5-a637-11b1b33a4c13","Type":"ContainerStarted","Data":"75ba8a1522f6541e38fecb16d6c3a6199d3256e055990f569feb76c3501650a9"}
Jul 17 05:25:17.626695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:17.626662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" event={"ID":"3b8864b9-1069-41e5-a637-11b1b33a4c13","Type":"ContainerStarted","Data":"2658451b1192b16febd9b628b5a3e344573e3c8cb8d718e38c6b5845503e1a3a"}
Jul 17 05:25:17.641654 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:17.641604    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" podStartSLOduration=1.641581913 podStartE2EDuration="1.641581913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:25:17.64085573 +0000 UTC m=+2121.377918083" watchObservedRunningTime="2026-07-17 05:25:17.641581913 +0000 UTC m=+2121.378644252"
Jul 17 05:25:18.632484 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:18.632449    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerID="75ba8a1522f6541e38fecb16d6c3a6199d3256e055990f569feb76c3501650a9" exitCode=7
Jul 17 05:25:18.632945 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:18.632545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" event={"ID":"3b8864b9-1069-41e5-a637-11b1b33a4c13","Type":"ContainerDied","Data":"75ba8a1522f6541e38fecb16d6c3a6199d3256e055990f569feb76c3501650a9"}
Jul 17 05:25:18.632945 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:18.632705    2582 scope.go:117] "RemoveContainer" containerID="75ba8a1522f6541e38fecb16d6c3a6199d3256e055990f569feb76c3501650a9"
Jul 17 05:25:19.639358 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:19.639310    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerID="2ef58d1350fbe5bb3568fcdbbb27ae61f1a41f5b6160dc5a6686f743069e3f8c" exitCode=7
Jul 17 05:25:19.639877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:19.639385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" event={"ID":"3b8864b9-1069-41e5-a637-11b1b33a4c13","Type":"ContainerDied","Data":"2ef58d1350fbe5bb3568fcdbbb27ae61f1a41f5b6160dc5a6686f743069e3f8c"}
Jul 17 05:25:19.639877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:19.639451    2582 scope.go:117] "RemoveContainer" containerID="75ba8a1522f6541e38fecb16d6c3a6199d3256e055990f569feb76c3501650a9"
Jul 17 05:25:19.639877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:19.639683    2582 scope.go:117] "RemoveContainer" containerID="2ef58d1350fbe5bb3568fcdbbb27ae61f1a41f5b6160dc5a6686f743069e3f8c"
Jul 17 05:25:19.640050 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:19.639921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-thph2_odh-ai-gateway-infra(3b8864b9-1069-41e5-a637-11b1b33a4c13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13"
Jul 17 05:25:20.645438 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:20.645409    2582 scope.go:117] "RemoveContainer" containerID="2ef58d1350fbe5bb3568fcdbbb27ae61f1a41f5b6160dc5a6686f743069e3f8c"
Jul 17 05:25:20.645834 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:20.645597    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-thph2_odh-ai-gateway-infra(3b8864b9-1069-41e5-a637-11b1b33a4c13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13"
Jul 17 05:25:32.724785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:32.724750    2582 scope.go:117] "RemoveContainer" containerID="2ef58d1350fbe5bb3568fcdbbb27ae61f1a41f5b6160dc5a6686f743069e3f8c"
Jul 17 05:25:33.701279 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:33.701223    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerID="76812ca8578a9926cee8d3f0fc89cedd13b6647a63c49a055dd3ba0826add14c" exitCode=7
Jul 17 05:25:33.701438 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:33.701297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" event={"ID":"3b8864b9-1069-41e5-a637-11b1b33a4c13","Type":"ContainerDied","Data":"76812ca8578a9926cee8d3f0fc89cedd13b6647a63c49a055dd3ba0826add14c"}
Jul 17 05:25:33.701438 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:33.701359    2582 scope.go:117] "RemoveContainer" containerID="2ef58d1350fbe5bb3568fcdbbb27ae61f1a41f5b6160dc5a6686f743069e3f8c"
Jul 17 05:25:33.701741 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:33.701716    2582 scope.go:117] "RemoveContainer" containerID="76812ca8578a9926cee8d3f0fc89cedd13b6647a63c49a055dd3ba0826add14c"
Jul 17 05:25:33.701943 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:33.701919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-thph2_odh-ai-gateway-infra(3b8864b9-1069-41e5-a637-11b1b33a4c13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13"
Jul 17 05:25:34.730041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:34.730009    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"]
Jul 17 05:25:34.863063 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:34.863037    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:34.960881 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:34.960840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8864b9-1069-41e5-a637-11b1b33a4c13-maas-api-service-ca\") pod \"3b8864b9-1069-41e5-a637-11b1b33a4c13\" (UID: \"3b8864b9-1069-41e5-a637-11b1b33a4c13\") "
Jul 17 05:25:34.961298 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:34.961239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8864b9-1069-41e5-a637-11b1b33a4c13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b8864b9-1069-41e5-a637-11b1b33a4c13" (UID: "3b8864b9-1069-41e5-a637-11b1b33a4c13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:25:35.062144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:35.062049    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8864b9-1069-41e5-a637-11b1b33a4c13-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:25:35.712093 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:35.712063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"
Jul 17 05:25:35.712093 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:35.712070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2" event={"ID":"3b8864b9-1069-41e5-a637-11b1b33a4c13","Type":"ContainerDied","Data":"2658451b1192b16febd9b628b5a3e344573e3c8cb8d718e38c6b5845503e1a3a"}
Jul 17 05:25:35.712344 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:35.712153    2582 scope.go:117] "RemoveContainer" containerID="76812ca8578a9926cee8d3f0fc89cedd13b6647a63c49a055dd3ba0826add14c"
Jul 17 05:25:35.731837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:35.731796    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"]
Jul 17 05:25:35.736889 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:35.736860    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-thph2"]
Jul 17 05:25:36.730504 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:36.730473    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" path="/var/lib/kubelet/pods/3b8864b9-1069-41e5-a637-11b1b33a4c13/volumes"
Jul 17 05:25:38.621900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.621868    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"]
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622168    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622184    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622196    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622203    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622277    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622286    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.622400 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.622293    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:38.626617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.626600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:38.629452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.629427    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:25:38.629625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.629467    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:25:38.632275 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.632234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"]
Jul 17 05:25:38.690484 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.690438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63e2-8eca-4ebd-ba42-c852e1ee2a28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t\" (UID: \"189b63e2-8eca-4ebd-ba42-c852e1ee2a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:38.791714 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.791677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63e2-8eca-4ebd-ba42-c852e1ee2a28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t\" (UID: \"189b63e2-8eca-4ebd-ba42-c852e1ee2a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:38.792316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.792294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63e2-8eca-4ebd-ba42-c852e1ee2a28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t\" (UID: \"189b63e2-8eca-4ebd-ba42-c852e1ee2a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:38.938454 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:38.938370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:39.073005 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:39.072837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"]
Jul 17 05:25:39.075646 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:25:39.075614    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189b63e2_8eca_4ebd_ba42_c852e1ee2a28.slice/crio-5d15be3a707c2e57171c1188a06848953176089f5a556fea691caa42496eb30b WatchSource:0}: Error finding container 5d15be3a707c2e57171c1188a06848953176089f5a556fea691caa42496eb30b: Status 404 returned error can't find the container with id 5d15be3a707c2e57171c1188a06848953176089f5a556fea691caa42496eb30b
Jul 17 05:25:39.731541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:39.731505    2582 generic.go:356] "Generic (PLEG): container finished" podID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerID="032060b027b19ef985de9227a217ea2ebd3ab722980ead064bc35438ee370b50" exitCode=6
Jul 17 05:25:39.731928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:39.731591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" event={"ID":"189b63e2-8eca-4ebd-ba42-c852e1ee2a28","Type":"ContainerDied","Data":"032060b027b19ef985de9227a217ea2ebd3ab722980ead064bc35438ee370b50"}
Jul 17 05:25:39.731928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:39.731647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" event={"ID":"189b63e2-8eca-4ebd-ba42-c852e1ee2a28","Type":"ContainerStarted","Data":"5d15be3a707c2e57171c1188a06848953176089f5a556fea691caa42496eb30b"}
Jul 17 05:25:39.731928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:39.731732    2582 scope.go:117] "RemoveContainer" containerID="032060b027b19ef985de9227a217ea2ebd3ab722980ead064bc35438ee370b50"
Jul 17 05:25:40.473040 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.473007    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"]
Jul 17 05:25:40.473351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.473338    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:40.473401 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.473353    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8864b9-1069-41e5-a637-11b1b33a4c13" containerName="revoke-keys"
Jul 17 05:25:40.476220 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.476202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:40.487541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.486596    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"]
Jul 17 05:25:40.507970 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.507930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d02f30-4c62-4e00-9175-7bcd7023e484-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-l7v55\" (UID: \"96d02f30-4c62-4e00-9175-7bcd7023e484\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:40.609197 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.609101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d02f30-4c62-4e00-9175-7bcd7023e484-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-l7v55\" (UID: \"96d02f30-4c62-4e00-9175-7bcd7023e484\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:40.609759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.609735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d02f30-4c62-4e00-9175-7bcd7023e484-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-l7v55\" (UID: \"96d02f30-4c62-4e00-9175-7bcd7023e484\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:40.737462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.737417    2582 generic.go:356] "Generic (PLEG): container finished" podID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerID="c316accac43ec3c5ec0dabde905949774b8a900fd8f7b46fd9f46d3f6e919767" exitCode=6
Jul 17 05:25:40.737859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.737496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" event={"ID":"189b63e2-8eca-4ebd-ba42-c852e1ee2a28","Type":"ContainerDied","Data":"c316accac43ec3c5ec0dabde905949774b8a900fd8f7b46fd9f46d3f6e919767"}
Jul 17 05:25:40.737859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.737555    2582 scope.go:117] "RemoveContainer" containerID="032060b027b19ef985de9227a217ea2ebd3ab722980ead064bc35438ee370b50"
Jul 17 05:25:40.737859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.737728    2582 scope.go:117] "RemoveContainer" containerID="c316accac43ec3c5ec0dabde905949774b8a900fd8f7b46fd9f46d3f6e919767"
Jul 17 05:25:40.737982 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:40.737942    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t_odh-ai-gateway-infra(189b63e2-8eca-4ebd-ba42-c852e1ee2a28)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28"
Jul 17 05:25:40.795156 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.795117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:40.932478 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:40.932449    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"]
Jul 17 05:25:40.935608 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:25:40.935578    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d02f30_4c62_4e00_9175_7bcd7023e484.slice/crio-2e486e62ca18dba28f8e196867822ee6e82b7caa910494c4420624a33f562ccc WatchSource:0}: Error finding container 2e486e62ca18dba28f8e196867822ee6e82b7caa910494c4420624a33f562ccc: Status 404 returned error can't find the container with id 2e486e62ca18dba28f8e196867822ee6e82b7caa910494c4420624a33f562ccc
Jul 17 05:25:41.742986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:41.742887    2582 generic.go:356] "Generic (PLEG): container finished" podID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerID="d8b1df5ec96a86f456e07ac114cc76df0ece0694d1e0f078751ed605d2ecfeb8" exitCode=6
Jul 17 05:25:41.742986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:41.742976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" event={"ID":"96d02f30-4c62-4e00-9175-7bcd7023e484","Type":"ContainerDied","Data":"d8b1df5ec96a86f456e07ac114cc76df0ece0694d1e0f078751ed605d2ecfeb8"}
Jul 17 05:25:41.743469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:41.743026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" event={"ID":"96d02f30-4c62-4e00-9175-7bcd7023e484","Type":"ContainerStarted","Data":"2e486e62ca18dba28f8e196867822ee6e82b7caa910494c4420624a33f562ccc"}
Jul 17 05:25:41.743469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:41.743148    2582 scope.go:117] "RemoveContainer" containerID="d8b1df5ec96a86f456e07ac114cc76df0ece0694d1e0f078751ed605d2ecfeb8"
Jul 17 05:25:41.744869 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:41.744851    2582 scope.go:117] "RemoveContainer" containerID="c316accac43ec3c5ec0dabde905949774b8a900fd8f7b46fd9f46d3f6e919767"
Jul 17 05:25:41.745017 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:41.744989    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t_odh-ai-gateway-infra(189b63e2-8eca-4ebd-ba42-c852e1ee2a28)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28"
Jul 17 05:25:42.750240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:42.750208    2582 generic.go:356] "Generic (PLEG): container finished" podID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerID="3ebcf74e002901a8164be740b69791761dfa68d67c8e317eec1bc9387db7c194" exitCode=6
Jul 17 05:25:42.750714 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:42.750282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" event={"ID":"96d02f30-4c62-4e00-9175-7bcd7023e484","Type":"ContainerDied","Data":"3ebcf74e002901a8164be740b69791761dfa68d67c8e317eec1bc9387db7c194"}
Jul 17 05:25:42.750714 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:42.750333    2582 scope.go:117] "RemoveContainer" containerID="d8b1df5ec96a86f456e07ac114cc76df0ece0694d1e0f078751ed605d2ecfeb8"
Jul 17 05:25:42.750714 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:42.750500    2582 scope.go:117] "RemoveContainer" containerID="3ebcf74e002901a8164be740b69791761dfa68d67c8e317eec1bc9387db7c194"
Jul 17 05:25:42.750714 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:42.750693    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-l7v55_odh-ai-gateway-infra(96d02f30-4c62-4e00-9175-7bcd7023e484)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484"
Jul 17 05:25:43.756249 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:43.756217    2582 scope.go:117] "RemoveContainer" containerID="3ebcf74e002901a8164be740b69791761dfa68d67c8e317eec1bc9387db7c194"
Jul 17 05:25:43.756702 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:43.756442    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-l7v55_odh-ai-gateway-infra(96d02f30-4c62-4e00-9175-7bcd7023e484)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484"
Jul 17 05:25:56.726986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:56.726902    2582 scope.go:117] "RemoveContainer" containerID="3ebcf74e002901a8164be740b69791761dfa68d67c8e317eec1bc9387db7c194"
Jul 17 05:25:56.726986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:56.726974    2582 scope.go:117] "RemoveContainer" containerID="c316accac43ec3c5ec0dabde905949774b8a900fd8f7b46fd9f46d3f6e919767"
Jul 17 05:25:57.813603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.813572    2582 generic.go:356] "Generic (PLEG): container finished" podID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerID="69144c8ec64db71f236da5f480f73ac0f79bc6af99672e6e03fc469d4455a755" exitCode=6
Jul 17 05:25:57.814100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.813654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" event={"ID":"96d02f30-4c62-4e00-9175-7bcd7023e484","Type":"ContainerDied","Data":"69144c8ec64db71f236da5f480f73ac0f79bc6af99672e6e03fc469d4455a755"}
Jul 17 05:25:57.814100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.813723    2582 scope.go:117] "RemoveContainer" containerID="3ebcf74e002901a8164be740b69791761dfa68d67c8e317eec1bc9387db7c194"
Jul 17 05:25:57.814100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.814037    2582 scope.go:117] "RemoveContainer" containerID="69144c8ec64db71f236da5f480f73ac0f79bc6af99672e6e03fc469d4455a755"
Jul 17 05:25:57.814322 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:57.814249    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-l7v55_odh-ai-gateway-infra(96d02f30-4c62-4e00-9175-7bcd7023e484)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484"
Jul 17 05:25:57.815662 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.815641    2582 generic.go:356] "Generic (PLEG): container finished" podID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerID="03e63f0a1c96c767885a364a3e489c34187d1710f5961d4ede352e73fc0d3e11" exitCode=6
Jul 17 05:25:57.815766 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.815682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" event={"ID":"189b63e2-8eca-4ebd-ba42-c852e1ee2a28","Type":"ContainerDied","Data":"03e63f0a1c96c767885a364a3e489c34187d1710f5961d4ede352e73fc0d3e11"}
Jul 17 05:25:57.815937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.815925    2582 scope.go:117] "RemoveContainer" containerID="03e63f0a1c96c767885a364a3e489c34187d1710f5961d4ede352e73fc0d3e11"
Jul 17 05:25:57.816095 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:25:57.816077    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t_odh-ai-gateway-infra(189b63e2-8eca-4ebd-ba42-c852e1ee2a28)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28"
Jul 17 05:25:57.824790 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:57.824768    2582 scope.go:117] "RemoveContainer" containerID="c316accac43ec3c5ec0dabde905949774b8a900fd8f7b46fd9f46d3f6e919767"
Jul 17 05:25:58.838446 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:58.838403    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"]
Jul 17 05:25:58.851108 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:58.851070    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"]
Jul 17 05:25:58.983439 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:58.983417    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:58.996385 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:58.996361    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:59.071464 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.071420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d02f30-4c62-4e00-9175-7bcd7023e484-maas-api-service-ca\") pod \"96d02f30-4c62-4e00-9175-7bcd7023e484\" (UID: \"96d02f30-4c62-4e00-9175-7bcd7023e484\") "
Jul 17 05:25:59.071681 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.071505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63e2-8eca-4ebd-ba42-c852e1ee2a28-maas-api-service-ca\") pod \"189b63e2-8eca-4ebd-ba42-c852e1ee2a28\" (UID: \"189b63e2-8eca-4ebd-ba42-c852e1ee2a28\") "
Jul 17 05:25:59.071859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.071833    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189b63e2-8eca-4ebd-ba42-c852e1ee2a28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "189b63e2-8eca-4ebd-ba42-c852e1ee2a28" (UID: "189b63e2-8eca-4ebd-ba42-c852e1ee2a28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:25:59.071930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.071841    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d02f30-4c62-4e00-9175-7bcd7023e484-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96d02f30-4c62-4e00-9175-7bcd7023e484" (UID: "96d02f30-4c62-4e00-9175-7bcd7023e484"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:25:59.172044 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.171949    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d02f30-4c62-4e00-9175-7bcd7023e484-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:25:59.172044 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.171984    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189b63e2-8eca-4ebd-ba42-c852e1ee2a28-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:25:59.828695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.828664    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"
Jul 17 05:25:59.828695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.828676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t" event={"ID":"189b63e2-8eca-4ebd-ba42-c852e1ee2a28","Type":"ContainerDied","Data":"5d15be3a707c2e57171c1188a06848953176089f5a556fea691caa42496eb30b"}
Jul 17 05:25:59.828928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.828746    2582 scope.go:117] "RemoveContainer" containerID="03e63f0a1c96c767885a364a3e489c34187d1710f5961d4ede352e73fc0d3e11"
Jul 17 05:25:59.830000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.829986    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"
Jul 17 05:25:59.830114 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.830033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55" event={"ID":"96d02f30-4c62-4e00-9175-7bcd7023e484","Type":"ContainerDied","Data":"2e486e62ca18dba28f8e196867822ee6e82b7caa910494c4420624a33f562ccc"}
Jul 17 05:25:59.838220 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.838202    2582 scope.go:117] "RemoveContainer" containerID="69144c8ec64db71f236da5f480f73ac0f79bc6af99672e6e03fc469d4455a755"
Jul 17 05:25:59.850673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.850647    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"]
Jul 17 05:25:59.858078 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.858054    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-qff8t"]
Jul 17 05:25:59.870835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.870809    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"]
Jul 17 05:25:59.876306 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:25:59.876281    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-l7v55"]
Jul 17 05:26:00.729439 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:00.729409    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" path="/var/lib/kubelet/pods/189b63e2-8eca-4ebd-ba42-c852e1ee2a28/volumes"
Jul 17 05:26:00.729749 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:00.729736    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" path="/var/lib/kubelet/pods/96d02f30-4c62-4e00-9175-7bcd7023e484/volumes"
Jul 17 05:26:06.731088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731061    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"]
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731331    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731342    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731351    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731357    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731364    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731369    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731379    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731384    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731389    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731395    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731445    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731453    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731458    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731463    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:06.731524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.731470    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:06.734514 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.734497    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:06.737319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.737290    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:26:06.737478 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.737320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:26:06.740475 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.740447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"]
Jul 17 05:26:06.837574 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.837535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d154eb6-eb9c-4c01-86f5-b6186d31566e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf\" (UID: \"7d154eb6-eb9c-4c01-86f5-b6186d31566e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:06.938726 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.938689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d154eb6-eb9c-4c01-86f5-b6186d31566e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf\" (UID: \"7d154eb6-eb9c-4c01-86f5-b6186d31566e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:06.939340 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:06.939320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d154eb6-eb9c-4c01-86f5-b6186d31566e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf\" (UID: \"7d154eb6-eb9c-4c01-86f5-b6186d31566e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:07.046975 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:07.046875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:07.187911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:07.187887    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"]
Jul 17 05:26:07.190530 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:26:07.190499    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d154eb6_eb9c_4c01_86f5_b6186d31566e.slice/crio-7251c5a93ab31c9130f56ee42bff1f44770a744461418cd074586f3291d2a315 WatchSource:0}: Error finding container 7251c5a93ab31c9130f56ee42bff1f44770a744461418cd074586f3291d2a315: Status 404 returned error can't find the container with id 7251c5a93ab31c9130f56ee42bff1f44770a744461418cd074586f3291d2a315
Jul 17 05:26:07.868713 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:07.868678    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerID="2b8fa8f8f1c205a21547309b3dc79c7890ed6b46cdb4ee3cf3c324246af4ed6d" exitCode=7
Jul 17 05:26:07.869274 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:07.868724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" event={"ID":"7d154eb6-eb9c-4c01-86f5-b6186d31566e","Type":"ContainerDied","Data":"2b8fa8f8f1c205a21547309b3dc79c7890ed6b46cdb4ee3cf3c324246af4ed6d"}
Jul 17 05:26:07.869274 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:07.868762    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" event={"ID":"7d154eb6-eb9c-4c01-86f5-b6186d31566e","Type":"ContainerStarted","Data":"7251c5a93ab31c9130f56ee42bff1f44770a744461418cd074586f3291d2a315"}
Jul 17 05:26:07.869274 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:07.868931    2582 scope.go:117] "RemoveContainer" containerID="2b8fa8f8f1c205a21547309b3dc79c7890ed6b46cdb4ee3cf3c324246af4ed6d"
Jul 17 05:26:08.874988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:08.874888    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerID="32f3957795ae4660510612edd058f7c884f10d53915e423645ce53664fed1295" exitCode=7
Jul 17 05:26:08.874988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:08.874973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" event={"ID":"7d154eb6-eb9c-4c01-86f5-b6186d31566e","Type":"ContainerDied","Data":"32f3957795ae4660510612edd058f7c884f10d53915e423645ce53664fed1295"}
Jul 17 05:26:08.875505 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:08.875051    2582 scope.go:117] "RemoveContainer" containerID="2b8fa8f8f1c205a21547309b3dc79c7890ed6b46cdb4ee3cf3c324246af4ed6d"
Jul 17 05:26:08.875505 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:08.875308    2582 scope.go:117] "RemoveContainer" containerID="32f3957795ae4660510612edd058f7c884f10d53915e423645ce53664fed1295"
Jul 17 05:26:08.875593 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:08.875513    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf_odh-ai-gateway-infra(7d154eb6-eb9c-4c01-86f5-b6186d31566e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e"
Jul 17 05:26:09.880700 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:09.880668    2582 scope.go:117] "RemoveContainer" containerID="32f3957795ae4660510612edd058f7c884f10d53915e423645ce53664fed1295"
Jul 17 05:26:09.881096 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:09.880871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf_odh-ai-gateway-infra(7d154eb6-eb9c-4c01-86f5-b6186d31566e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e"
Jul 17 05:26:21.725214 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:21.725172    2582 scope.go:117] "RemoveContainer" containerID="32f3957795ae4660510612edd058f7c884f10d53915e423645ce53664fed1295"
Jul 17 05:26:21.931915 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:21.931878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" event={"ID":"7d154eb6-eb9c-4c01-86f5-b6186d31566e","Type":"ContainerStarted","Data":"016fa339b732424c231e8e8c8bf9fa570d58e272babeb4d08067f36cee9881d1"}
Jul 17 05:26:21.948661 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:21.948597    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" podStartSLOduration=15.94857953 podStartE2EDuration="15.94857953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:26:21.946827112 +0000 UTC m=+2185.683889451" watchObservedRunningTime="2026-07-17 05:26:21.94857953 +0000 UTC m=+2185.685641866"
Jul 17 05:26:22.937126 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:22.937092    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerID="016fa339b732424c231e8e8c8bf9fa570d58e272babeb4d08067f36cee9881d1" exitCode=7
Jul 17 05:26:22.937722 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:22.937173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" event={"ID":"7d154eb6-eb9c-4c01-86f5-b6186d31566e","Type":"ContainerDied","Data":"016fa339b732424c231e8e8c8bf9fa570d58e272babeb4d08067f36cee9881d1"}
Jul 17 05:26:22.937722 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:22.937274    2582 scope.go:117] "RemoveContainer" containerID="32f3957795ae4660510612edd058f7c884f10d53915e423645ce53664fed1295"
Jul 17 05:26:22.937722 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:22.937532    2582 scope.go:117] "RemoveContainer" containerID="016fa339b732424c231e8e8c8bf9fa570d58e272babeb4d08067f36cee9881d1"
Jul 17 05:26:22.937891 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:22.937722    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf_odh-ai-gateway-infra(7d154eb6-eb9c-4c01-86f5-b6186d31566e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e"
Jul 17 05:26:22.959123 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:22.959086    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"]
Jul 17 05:26:24.076520 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.076493    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:24.078379 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.078358    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d154eb6-eb9c-4c01-86f5-b6186d31566e-maas-api-service-ca\") pod \"7d154eb6-eb9c-4c01-86f5-b6186d31566e\" (UID: \"7d154eb6-eb9c-4c01-86f5-b6186d31566e\") "
Jul 17 05:26:24.078685 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.078666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d154eb6-eb9c-4c01-86f5-b6186d31566e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d154eb6-eb9c-4c01-86f5-b6186d31566e" (UID: "7d154eb6-eb9c-4c01-86f5-b6186d31566e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:26:24.178984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.178933    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d154eb6-eb9c-4c01-86f5-b6186d31566e-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:26:24.948590 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.948558    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"
Jul 17 05:26:24.948784 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.948555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf" event={"ID":"7d154eb6-eb9c-4c01-86f5-b6186d31566e","Type":"ContainerDied","Data":"7251c5a93ab31c9130f56ee42bff1f44770a744461418cd074586f3291d2a315"}
Jul 17 05:26:24.948784 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.948710    2582 scope.go:117] "RemoveContainer" containerID="016fa339b732424c231e8e8c8bf9fa570d58e272babeb4d08067f36cee9881d1"
Jul 17 05:26:24.965127 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.965101    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"]
Jul 17 05:26:24.972468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:24.972446    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-js9gf"]
Jul 17 05:26:26.730418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:26.730387    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" path="/var/lib/kubelet/pods/7d154eb6-eb9c-4c01-86f5-b6186d31566e/volumes"
Jul 17 05:26:30.518622 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518584    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"]
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518896    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518907    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518917    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518923    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518929    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518935    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d02f30-4c62-4e00-9175-7bcd7023e484" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518944    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518949    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518991    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.518998    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.519004    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d154eb6-eb9c-4c01-86f5-b6186d31566e" containerName="revoke-keys"
Jul 17 05:26:30.519016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.519011    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="189b63e2-8eca-4ebd-ba42-c852e1ee2a28" containerName="revoke-keys"
Jul 17 05:26:30.521759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.521741    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:30.524216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.524189    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:26:30.524352 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.524194    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:26:30.528582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.528545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"]
Jul 17 05:26:30.624148 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.624116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b28a0c-d922-4428-a156-a9decce10ff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-9v9fd\" (UID: \"75b28a0c-d922-4428-a156-a9decce10ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:30.725026 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.724984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b28a0c-d922-4428-a156-a9decce10ff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-9v9fd\" (UID: \"75b28a0c-d922-4428-a156-a9decce10ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:30.725599 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.725580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b28a0c-d922-4428-a156-a9decce10ff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-9v9fd\" (UID: \"75b28a0c-d922-4428-a156-a9decce10ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:30.833939 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.833859    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:30.962664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.962630    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"]
Jul 17 05:26:30.966456 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:26:30.966419    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b28a0c_d922_4428_a156_a9decce10ff3.slice/crio-71fe24672881f5155cec23feec04c227e700c8db00f4fe699fdfdeb194b4a860 WatchSource:0}: Error finding container 71fe24672881f5155cec23feec04c227e700c8db00f4fe699fdfdeb194b4a860: Status 404 returned error can't find the container with id 71fe24672881f5155cec23feec04c227e700c8db00f4fe699fdfdeb194b4a860
Jul 17 05:26:30.976106 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:30.976070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" event={"ID":"75b28a0c-d922-4428-a156-a9decce10ff3","Type":"ContainerStarted","Data":"71fe24672881f5155cec23feec04c227e700c8db00f4fe699fdfdeb194b4a860"}
Jul 17 05:26:31.981278 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:31.981231    2582 generic.go:356] "Generic (PLEG): container finished" podID="75b28a0c-d922-4428-a156-a9decce10ff3" containerID="b880007b7e9ee01422aa1f900463f3df16dcfff774ff1e26e6748c97f3cc3e56" exitCode=6
Jul 17 05:26:31.981651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:31.981296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" event={"ID":"75b28a0c-d922-4428-a156-a9decce10ff3","Type":"ContainerDied","Data":"b880007b7e9ee01422aa1f900463f3df16dcfff774ff1e26e6748c97f3cc3e56"}
Jul 17 05:26:31.981651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:31.981514    2582 scope.go:117] "RemoveContainer" containerID="b880007b7e9ee01422aa1f900463f3df16dcfff774ff1e26e6748c97f3cc3e56"
Jul 17 05:26:32.987005 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:32.986976    2582 generic.go:356] "Generic (PLEG): container finished" podID="75b28a0c-d922-4428-a156-a9decce10ff3" containerID="11a13e7c6f3ff664becb56d2fe1608903f3d0f123996346a2b78ebf05870fa67" exitCode=6
Jul 17 05:26:32.987419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:32.987052    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" event={"ID":"75b28a0c-d922-4428-a156-a9decce10ff3","Type":"ContainerDied","Data":"11a13e7c6f3ff664becb56d2fe1608903f3d0f123996346a2b78ebf05870fa67"}
Jul 17 05:26:32.987419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:32.987093    2582 scope.go:117] "RemoveContainer" containerID="b880007b7e9ee01422aa1f900463f3df16dcfff774ff1e26e6748c97f3cc3e56"
Jul 17 05:26:32.987419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:32.987233    2582 scope.go:117] "RemoveContainer" containerID="11a13e7c6f3ff664becb56d2fe1608903f3d0f123996346a2b78ebf05870fa67"
Jul 17 05:26:32.987549 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:32.987478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-9v9fd_odh-ai-gateway-infra(75b28a0c-d922-4428-a156-a9decce10ff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" podUID="75b28a0c-d922-4428-a156-a9decce10ff3"
Jul 17 05:26:33.993227 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:33.993201    2582 scope.go:117] "RemoveContainer" containerID="11a13e7c6f3ff664becb56d2fe1608903f3d0f123996346a2b78ebf05870fa67"
Jul 17 05:26:33.995609 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:33.993400    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-9v9fd_odh-ai-gateway-infra(75b28a0c-d922-4428-a156-a9decce10ff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" podUID="75b28a0c-d922-4428-a156-a9decce10ff3"
Jul 17 05:26:38.667181 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.667145    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"]
Jul 17 05:26:38.670374 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.670356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:38.677795 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.677517    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"]
Jul 17 05:26:38.785572 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.785527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f3f89b-d660-4a41-8813-6f326b08151c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql\" (UID: \"b3f3f89b-d660-4a41-8813-6f326b08151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:38.886921 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.886882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f3f89b-d660-4a41-8813-6f326b08151c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql\" (UID: \"b3f3f89b-d660-4a41-8813-6f326b08151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:38.887582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.887559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f3f89b-d660-4a41-8813-6f326b08151c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql\" (UID: \"b3f3f89b-d660-4a41-8813-6f326b08151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:38.982515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:38.982473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:39.120163 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:39.120142    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"]
Jul 17 05:26:39.122355 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:26:39.122328    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f3f89b_d660_4a41_8813_6f326b08151c.slice/crio-45c107652f8de6dc85bdbf935b5e00e51e20f83a65ce08f60d9093a231016d09 WatchSource:0}: Error finding container 45c107652f8de6dc85bdbf935b5e00e51e20f83a65ce08f60d9093a231016d09: Status 404 returned error can't find the container with id 45c107652f8de6dc85bdbf935b5e00e51e20f83a65ce08f60d9093a231016d09
Jul 17 05:26:40.018581 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:40.018486    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerID="3821cd99fc034370554f2ba78955f174d50694ae2b2b02808a13fb73b84832aa" exitCode=6
Jul 17 05:26:40.018581 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:40.018552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" event={"ID":"b3f3f89b-d660-4a41-8813-6f326b08151c","Type":"ContainerDied","Data":"3821cd99fc034370554f2ba78955f174d50694ae2b2b02808a13fb73b84832aa"}
Jul 17 05:26:40.019096 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:40.018617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" event={"ID":"b3f3f89b-d660-4a41-8813-6f326b08151c","Type":"ContainerStarted","Data":"45c107652f8de6dc85bdbf935b5e00e51e20f83a65ce08f60d9093a231016d09"}
Jul 17 05:26:40.019096 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:40.018818    2582 scope.go:117] "RemoveContainer" containerID="3821cd99fc034370554f2ba78955f174d50694ae2b2b02808a13fb73b84832aa"
Jul 17 05:26:41.025061 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:41.025026    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerID="4a378361a9fa51e6fdc50f161ce3ec4afd0a8e9a5459a7ecb29f672930ebb896" exitCode=6
Jul 17 05:26:41.025577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:41.025105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" event={"ID":"b3f3f89b-d660-4a41-8813-6f326b08151c","Type":"ContainerDied","Data":"4a378361a9fa51e6fdc50f161ce3ec4afd0a8e9a5459a7ecb29f672930ebb896"}
Jul 17 05:26:41.025577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:41.025148    2582 scope.go:117] "RemoveContainer" containerID="3821cd99fc034370554f2ba78955f174d50694ae2b2b02808a13fb73b84832aa"
Jul 17 05:26:41.025577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:41.025423    2582 scope.go:117] "RemoveContainer" containerID="4a378361a9fa51e6fdc50f161ce3ec4afd0a8e9a5459a7ecb29f672930ebb896"
Jul 17 05:26:41.025752 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:41.025664    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql_odh-ai-gateway-infra(b3f3f89b-d660-4a41-8813-6f326b08151c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c"
Jul 17 05:26:42.030593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:42.030568    2582 scope.go:117] "RemoveContainer" containerID="4a378361a9fa51e6fdc50f161ce3ec4afd0a8e9a5459a7ecb29f672930ebb896"
Jul 17 05:26:42.031032 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:42.030747    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql_odh-ai-gateway-infra(b3f3f89b-d660-4a41-8813-6f326b08151c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c"
Jul 17 05:26:46.728090 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:46.728054    2582 scope.go:117] "RemoveContainer" containerID="11a13e7c6f3ff664becb56d2fe1608903f3d0f123996346a2b78ebf05870fa67"
Jul 17 05:26:47.054161 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:47.054072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" event={"ID":"75b28a0c-d922-4428-a156-a9decce10ff3","Type":"ContainerStarted","Data":"70ff595a1e3dc710d067fcb8a58f1e31717edc597e9b99dca855074cdb7cc10b"}
Jul 17 05:26:47.072496 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:47.072444    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" podStartSLOduration=17.072423646 podStartE2EDuration="17.072423646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:26:47.070931242 +0000 UTC m=+2210.807993580" watchObservedRunningTime="2026-07-17 05:26:47.072423646 +0000 UTC m=+2210.809485985"
Jul 17 05:26:48.059626 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:48.059592    2582 generic.go:356] "Generic (PLEG): container finished" podID="75b28a0c-d922-4428-a156-a9decce10ff3" containerID="70ff595a1e3dc710d067fcb8a58f1e31717edc597e9b99dca855074cdb7cc10b" exitCode=6
Jul 17 05:26:48.060070 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:48.059662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" event={"ID":"75b28a0c-d922-4428-a156-a9decce10ff3","Type":"ContainerDied","Data":"70ff595a1e3dc710d067fcb8a58f1e31717edc597e9b99dca855074cdb7cc10b"}
Jul 17 05:26:48.060070 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:48.059718    2582 scope.go:117] "RemoveContainer" containerID="11a13e7c6f3ff664becb56d2fe1608903f3d0f123996346a2b78ebf05870fa67"
Jul 17 05:26:48.060070 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:48.059981    2582 scope.go:117] "RemoveContainer" containerID="70ff595a1e3dc710d067fcb8a58f1e31717edc597e9b99dca855074cdb7cc10b"
Jul 17 05:26:48.060226 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:48.060200    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-9v9fd_odh-ai-gateway-infra(75b28a0c-d922-4428-a156-a9decce10ff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" podUID="75b28a0c-d922-4428-a156-a9decce10ff3"
Jul 17 05:26:48.080390 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:48.080368    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"]
Jul 17 05:26:49.194540 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:49.194515    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:49.274635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:49.274589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b28a0c-d922-4428-a156-a9decce10ff3-maas-api-service-ca\") pod \"75b28a0c-d922-4428-a156-a9decce10ff3\" (UID: \"75b28a0c-d922-4428-a156-a9decce10ff3\") "
Jul 17 05:26:49.274977 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:49.274955    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b28a0c-d922-4428-a156-a9decce10ff3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75b28a0c-d922-4428-a156-a9decce10ff3" (UID: "75b28a0c-d922-4428-a156-a9decce10ff3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:26:49.375972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:49.375885    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b28a0c-d922-4428-a156-a9decce10ff3-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:26:50.070162 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:50.070135    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"
Jul 17 05:26:50.070364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:50.070161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd" event={"ID":"75b28a0c-d922-4428-a156-a9decce10ff3","Type":"ContainerDied","Data":"71fe24672881f5155cec23feec04c227e700c8db00f4fe699fdfdeb194b4a860"}
Jul 17 05:26:50.070364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:50.070215    2582 scope.go:117] "RemoveContainer" containerID="70ff595a1e3dc710d067fcb8a58f1e31717edc597e9b99dca855074cdb7cc10b"
Jul 17 05:26:50.090832 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:50.090798    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"]
Jul 17 05:26:50.095834 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:50.095811    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-9v9fd"]
Jul 17 05:26:50.729397 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:50.729359    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" path="/var/lib/kubelet/pods/75b28a0c-d922-4428-a156-a9decce10ff3/volumes"
Jul 17 05:26:53.725207 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:53.725173    2582 scope.go:117] "RemoveContainer" containerID="4a378361a9fa51e6fdc50f161ce3ec4afd0a8e9a5459a7ecb29f672930ebb896"
Jul 17 05:26:54.089134 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:54.089098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" event={"ID":"b3f3f89b-d660-4a41-8813-6f326b08151c","Type":"ContainerStarted","Data":"df881d6d6ccdaba227ad59910636074f118f751c019212c2d96171090cb5b029"}
Jul 17 05:26:54.105460 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:54.105407    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" podStartSLOduration=16.10539205 podStartE2EDuration="16.10539205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:26:54.104250192 +0000 UTC m=+2217.841312552" watchObservedRunningTime="2026-07-17 05:26:54.10539205 +0000 UTC m=+2217.842454386"
Jul 17 05:26:55.095548 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:55.095509    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerID="df881d6d6ccdaba227ad59910636074f118f751c019212c2d96171090cb5b029" exitCode=6
Jul 17 05:26:55.095972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:55.095573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" event={"ID":"b3f3f89b-d660-4a41-8813-6f326b08151c","Type":"ContainerDied","Data":"df881d6d6ccdaba227ad59910636074f118f751c019212c2d96171090cb5b029"}
Jul 17 05:26:55.095972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:55.095635    2582 scope.go:117] "RemoveContainer" containerID="4a378361a9fa51e6fdc50f161ce3ec4afd0a8e9a5459a7ecb29f672930ebb896"
Jul 17 05:26:55.095972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:55.095917    2582 scope.go:117] "RemoveContainer" containerID="df881d6d6ccdaba227ad59910636074f118f751c019212c2d96171090cb5b029"
Jul 17 05:26:55.096155 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:55.096098    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql_odh-ai-gateway-infra(b3f3f89b-d660-4a41-8813-6f326b08151c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c"
Jul 17 05:26:55.114007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:55.113979    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"]
Jul 17 05:26:56.230838 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.230810    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:56.334067 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.334029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f3f89b-d660-4a41-8813-6f326b08151c-maas-api-service-ca\") pod \"b3f3f89b-d660-4a41-8813-6f326b08151c\" (UID: \"b3f3f89b-d660-4a41-8813-6f326b08151c\") "
Jul 17 05:26:56.334454 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.334434    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f3f89b-d660-4a41-8813-6f326b08151c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3f3f89b-d660-4a41-8813-6f326b08151c" (UID: "b3f3f89b-d660-4a41-8813-6f326b08151c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:26:56.435002 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.434914    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f3f89b-d660-4a41-8813-6f326b08151c-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:26:56.768662 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768619    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"]
Jul 17 05:26:56.768941 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768927    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768944    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768952    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768957    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768969    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768975    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768981    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:26:56.768989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.768986    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:26:56.769200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.769034    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:26:56.769200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.769040    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:26:56.769200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.769045    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:26:56.769200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.769051    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:26:56.771903 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.771884    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:26:56.778625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.778598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"]
Jul 17 05:26:56.837986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.837939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d6b11-d4aa-42e1-9870-978fe8db8f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r\" (UID: \"946d6b11-d4aa-42e1-9870-978fe8db8f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:26:56.938656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.938631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d6b11-d4aa-42e1-9870-978fe8db8f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r\" (UID: \"946d6b11-d4aa-42e1-9870-978fe8db8f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:26:56.939324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:56.939280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d6b11-d4aa-42e1-9870-978fe8db8f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r\" (UID: \"946d6b11-d4aa-42e1-9870-978fe8db8f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:26:57.083183 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.083089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:26:57.108026 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.107999    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"
Jul 17 05:26:57.108026 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.108010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql" event={"ID":"b3f3f89b-d660-4a41-8813-6f326b08151c","Type":"ContainerDied","Data":"45c107652f8de6dc85bdbf935b5e00e51e20f83a65ce08f60d9093a231016d09"}
Jul 17 05:26:57.108287 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.108084    2582 scope.go:117] "RemoveContainer" containerID="df881d6d6ccdaba227ad59910636074f118f751c019212c2d96171090cb5b029"
Jul 17 05:26:57.123884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.123853    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"]
Jul 17 05:26:57.125480 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.125451    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-55mql"]
Jul 17 05:26:57.218702 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:57.218673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"]
Jul 17 05:26:57.221966 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:26:57.221925    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946d6b11_d4aa_42e1_9870_978fe8db8f7a.slice/crio-fdd26cd8d64f6c7c9c1e3b0a6ea2cf929a0c5481aafabce9064cfe94f8965d5d WatchSource:0}: Error finding container fdd26cd8d64f6c7c9c1e3b0a6ea2cf929a0c5481aafabce9064cfe94f8965d5d: Status 404 returned error can't find the container with id fdd26cd8d64f6c7c9c1e3b0a6ea2cf929a0c5481aafabce9064cfe94f8965d5d
Jul 17 05:26:58.114733 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:58.114641    2582 generic.go:356] "Generic (PLEG): container finished" podID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerID="d3f8bfd7784c29e267dba9cf198044ff78de5f76e99268cc1d16b000684d3f9e" exitCode=7
Jul 17 05:26:58.115170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:58.114727    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" event={"ID":"946d6b11-d4aa-42e1-9870-978fe8db8f7a","Type":"ContainerDied","Data":"d3f8bfd7784c29e267dba9cf198044ff78de5f76e99268cc1d16b000684d3f9e"}
Jul 17 05:26:58.115170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:58.114785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" event={"ID":"946d6b11-d4aa-42e1-9870-978fe8db8f7a","Type":"ContainerStarted","Data":"fdd26cd8d64f6c7c9c1e3b0a6ea2cf929a0c5481aafabce9064cfe94f8965d5d"}
Jul 17 05:26:58.115170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:58.114922    2582 scope.go:117] "RemoveContainer" containerID="d3f8bfd7784c29e267dba9cf198044ff78de5f76e99268cc1d16b000684d3f9e"
Jul 17 05:26:58.729678 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:58.729644    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" path="/var/lib/kubelet/pods/b3f3f89b-d660-4a41-8813-6f326b08151c/volumes"
Jul 17 05:26:59.127282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:59.127228    2582 generic.go:356] "Generic (PLEG): container finished" podID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerID="ece6c3c1b5d39c77dd1cb39a4845b83730162bc0b18bc34af1905cb319bd74e2" exitCode=7
Jul 17 05:26:59.127755 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:59.127310    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" event={"ID":"946d6b11-d4aa-42e1-9870-978fe8db8f7a","Type":"ContainerDied","Data":"ece6c3c1b5d39c77dd1cb39a4845b83730162bc0b18bc34af1905cb319bd74e2"}
Jul 17 05:26:59.127755 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:59.127391    2582 scope.go:117] "RemoveContainer" containerID="d3f8bfd7784c29e267dba9cf198044ff78de5f76e99268cc1d16b000684d3f9e"
Jul 17 05:26:59.127755 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:26:59.127629    2582 scope.go:117] "RemoveContainer" containerID="ece6c3c1b5d39c77dd1cb39a4845b83730162bc0b18bc34af1905cb319bd74e2"
Jul 17 05:26:59.127882 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:26:59.127798    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r_odh-ai-gateway-infra(946d6b11-d4aa-42e1-9870-978fe8db8f7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a"
Jul 17 05:27:00.132944 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:00.132918    2582 scope.go:117] "RemoveContainer" containerID="ece6c3c1b5d39c77dd1cb39a4845b83730162bc0b18bc34af1905cb319bd74e2"
Jul 17 05:27:00.133357 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:00.133118    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r_odh-ai-gateway-infra(946d6b11-d4aa-42e1-9870-978fe8db8f7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a"
Jul 17 05:27:13.725379 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:13.725350    2582 scope.go:117] "RemoveContainer" containerID="ece6c3c1b5d39c77dd1cb39a4845b83730162bc0b18bc34af1905cb319bd74e2"
Jul 17 05:27:14.190204 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:14.190166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" event={"ID":"946d6b11-d4aa-42e1-9870-978fe8db8f7a","Type":"ContainerStarted","Data":"148e5faa4b421c8afecebcf99244819106104db732a71c3313005f894090c56f"}
Jul 17 05:27:14.205810 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:14.205759    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" podStartSLOduration=18.205744421 podStartE2EDuration="18.205744421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:27:14.204386219 +0000 UTC m=+2237.941448555" watchObservedRunningTime="2026-07-17 05:27:14.205744421 +0000 UTC m=+2237.942806755"
Jul 17 05:27:15.195768 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:15.195737    2582 generic.go:356] "Generic (PLEG): container finished" podID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerID="148e5faa4b421c8afecebcf99244819106104db732a71c3313005f894090c56f" exitCode=7
Jul 17 05:27:15.196224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:15.195776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" event={"ID":"946d6b11-d4aa-42e1-9870-978fe8db8f7a","Type":"ContainerDied","Data":"148e5faa4b421c8afecebcf99244819106104db732a71c3313005f894090c56f"}
Jul 17 05:27:15.196224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:15.195838    2582 scope.go:117] "RemoveContainer" containerID="ece6c3c1b5d39c77dd1cb39a4845b83730162bc0b18bc34af1905cb319bd74e2"
Jul 17 05:27:15.196224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:15.196098    2582 scope.go:117] "RemoveContainer" containerID="148e5faa4b421c8afecebcf99244819106104db732a71c3313005f894090c56f"
Jul 17 05:27:15.196426 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:15.196352    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r_odh-ai-gateway-infra(946d6b11-d4aa-42e1-9870-978fe8db8f7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a"
Jul 17 05:27:15.219230 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:15.219193    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"]
Jul 17 05:27:16.327555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:16.327523    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:27:16.506095 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:16.506004    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d6b11-d4aa-42e1-9870-978fe8db8f7a-maas-api-service-ca\") pod \"946d6b11-d4aa-42e1-9870-978fe8db8f7a\" (UID: \"946d6b11-d4aa-42e1-9870-978fe8db8f7a\") "
Jul 17 05:27:16.506419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:16.506400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946d6b11-d4aa-42e1-9870-978fe8db8f7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "946d6b11-d4aa-42e1-9870-978fe8db8f7a" (UID: "946d6b11-d4aa-42e1-9870-978fe8db8f7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:27:16.606905 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:16.606865    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946d6b11-d4aa-42e1-9870-978fe8db8f7a-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:27:17.206797 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:17.206755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r" event={"ID":"946d6b11-d4aa-42e1-9870-978fe8db8f7a","Type":"ContainerDied","Data":"fdd26cd8d64f6c7c9c1e3b0a6ea2cf929a0c5481aafabce9064cfe94f8965d5d"}
Jul 17 05:27:17.206797 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:17.206790    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"
Jul 17 05:27:17.207035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:17.206828    2582 scope.go:117] "RemoveContainer" containerID="148e5faa4b421c8afecebcf99244819106104db732a71c3313005f894090c56f"
Jul 17 05:27:17.222070 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:17.222041    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"]
Jul 17 05:27:17.226634 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:17.226607    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-sgb8r"]
Jul 17 05:27:18.729453 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:18.729423    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" path="/var/lib/kubelet/pods/946d6b11-d4aa-42e1-9870-978fe8db8f7a/volumes"
Jul 17 05:27:20.567863 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.567828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"]
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568120    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568132    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568145    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568150    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568158    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568166    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568178    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568183    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568225    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b28a0c-d922-4428-a156-a9decce10ff3" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568233    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568238    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568244    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:20.568656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.568251    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f3f89b-d660-4a41-8813-6f326b08151c" containerName="revoke-keys"
Jul 17 05:27:20.573846 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.572161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:20.575334 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.575310    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:27:20.575464 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.575314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:27:20.578239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.578218    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"]
Jul 17 05:27:20.739719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.739683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73278d02-59dd-4fc8-a2d9-449fce28d880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-bzmnw\" (UID: \"73278d02-59dd-4fc8-a2d9-449fce28d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:20.841086 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.840998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73278d02-59dd-4fc8-a2d9-449fce28d880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-bzmnw\" (UID: \"73278d02-59dd-4fc8-a2d9-449fce28d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:20.841655 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.841637    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73278d02-59dd-4fc8-a2d9-449fce28d880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-bzmnw\" (UID: \"73278d02-59dd-4fc8-a2d9-449fce28d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:20.885044 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:20.885015    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:21.013121 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:21.013075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"]
Jul 17 05:27:21.015918 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:27:21.015877    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73278d02_59dd_4fc8_a2d9_449fce28d880.slice/crio-2c95994be4c3b3efee2ac93ce6a69a385b20eb96d0cb1e0e00fcf1e64def90a4 WatchSource:0}: Error finding container 2c95994be4c3b3efee2ac93ce6a69a385b20eb96d0cb1e0e00fcf1e64def90a4: Status 404 returned error can't find the container with id 2c95994be4c3b3efee2ac93ce6a69a385b20eb96d0cb1e0e00fcf1e64def90a4
Jul 17 05:27:21.225444 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:21.225412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" event={"ID":"73278d02-59dd-4fc8-a2d9-449fce28d880","Type":"ContainerStarted","Data":"2c95994be4c3b3efee2ac93ce6a69a385b20eb96d0cb1e0e00fcf1e64def90a4"}
Jul 17 05:27:22.230482 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:22.230445    2582 generic.go:356] "Generic (PLEG): container finished" podID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerID="007195610b819f880fde7fbd0295351e9c51d71ddd053b7311678b79be2bfb21" exitCode=6
Jul 17 05:27:22.230898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:22.230500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" event={"ID":"73278d02-59dd-4fc8-a2d9-449fce28d880","Type":"ContainerDied","Data":"007195610b819f880fde7fbd0295351e9c51d71ddd053b7311678b79be2bfb21"}
Jul 17 05:27:22.230898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:22.230734    2582 scope.go:117] "RemoveContainer" containerID="007195610b819f880fde7fbd0295351e9c51d71ddd053b7311678b79be2bfb21"
Jul 17 05:27:23.237039 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:23.237005    2582 generic.go:356] "Generic (PLEG): container finished" podID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerID="116a0ff87a0bb4fb3dc842a45566fc845f96bd4b68eb1f2563bdca072268cdb3" exitCode=6
Jul 17 05:27:23.237452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:23.237095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" event={"ID":"73278d02-59dd-4fc8-a2d9-449fce28d880","Type":"ContainerDied","Data":"116a0ff87a0bb4fb3dc842a45566fc845f96bd4b68eb1f2563bdca072268cdb3"}
Jul 17 05:27:23.237452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:23.237155    2582 scope.go:117] "RemoveContainer" containerID="007195610b819f880fde7fbd0295351e9c51d71ddd053b7311678b79be2bfb21"
Jul 17 05:27:23.237452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:23.237366    2582 scope.go:117] "RemoveContainer" containerID="116a0ff87a0bb4fb3dc842a45566fc845f96bd4b68eb1f2563bdca072268cdb3"
Jul 17 05:27:23.237646 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:23.237574    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-bzmnw_odh-ai-gateway-infra(73278d02-59dd-4fc8-a2d9-449fce28d880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880"
Jul 17 05:27:24.243976 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:24.243952    2582 scope.go:117] "RemoveContainer" containerID="116a0ff87a0bb4fb3dc842a45566fc845f96bd4b68eb1f2563bdca072268cdb3"
Jul 17 05:27:24.244497 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:24.244167    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-bzmnw_odh-ai-gateway-infra(73278d02-59dd-4fc8-a2d9-449fce28d880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880"
Jul 17 05:27:28.710242 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.710204    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"]
Jul 17 05:27:28.710630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.710535    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:28.710630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.710547    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="946d6b11-d4aa-42e1-9870-978fe8db8f7a" containerName="revoke-keys"
Jul 17 05:27:28.714726 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.714706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:28.720732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.720700    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"]
Jul 17 05:27:28.801029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.800993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f72ce-5c74-49c3-b7f1-daf5e4547851-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz\" (UID: \"0e7f72ce-5c74-49c3-b7f1-daf5e4547851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:28.902402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.902297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f72ce-5c74-49c3-b7f1-daf5e4547851-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz\" (UID: \"0e7f72ce-5c74-49c3-b7f1-daf5e4547851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:28.902933 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:28.902909    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f72ce-5c74-49c3-b7f1-daf5e4547851-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz\" (UID: \"0e7f72ce-5c74-49c3-b7f1-daf5e4547851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:29.027233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:29.027204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:29.159869 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:29.159781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"]
Jul 17 05:27:29.166159 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:27:29.165218    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7f72ce_5c74_49c3_b7f1_daf5e4547851.slice/crio-aa2a227173e5cc45ea7290d0a66b8b828a47e3fc0de7eed2610c29bd0593b226 WatchSource:0}: Error finding container aa2a227173e5cc45ea7290d0a66b8b828a47e3fc0de7eed2610c29bd0593b226: Status 404 returned error can't find the container with id aa2a227173e5cc45ea7290d0a66b8b828a47e3fc0de7eed2610c29bd0593b226
Jul 17 05:27:29.266498 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:29.266454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" event={"ID":"0e7f72ce-5c74-49c3-b7f1-daf5e4547851","Type":"ContainerStarted","Data":"aa2a227173e5cc45ea7290d0a66b8b828a47e3fc0de7eed2610c29bd0593b226"}
Jul 17 05:27:30.272144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:30.272108    2582 generic.go:356] "Generic (PLEG): container finished" podID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerID="1991362adcc1a1d0ac3834cef6d95dc3cb6b62bc2af47f77d7e36f565f18bee0" exitCode=6
Jul 17 05:27:30.272583 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:30.272191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" event={"ID":"0e7f72ce-5c74-49c3-b7f1-daf5e4547851","Type":"ContainerDied","Data":"1991362adcc1a1d0ac3834cef6d95dc3cb6b62bc2af47f77d7e36f565f18bee0"}
Jul 17 05:27:30.272583 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:30.272408    2582 scope.go:117] "RemoveContainer" containerID="1991362adcc1a1d0ac3834cef6d95dc3cb6b62bc2af47f77d7e36f565f18bee0"
Jul 17 05:27:31.277862 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:31.277828    2582 generic.go:356] "Generic (PLEG): container finished" podID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerID="a45356ee2281b4aa3092415a3ee765b5438728acc3b952c9b94425c28afafd52" exitCode=6
Jul 17 05:27:31.278331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:31.277919    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" event={"ID":"0e7f72ce-5c74-49c3-b7f1-daf5e4547851","Type":"ContainerDied","Data":"a45356ee2281b4aa3092415a3ee765b5438728acc3b952c9b94425c28afafd52"}
Jul 17 05:27:31.278331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:31.277988    2582 scope.go:117] "RemoveContainer" containerID="1991362adcc1a1d0ac3834cef6d95dc3cb6b62bc2af47f77d7e36f565f18bee0"
Jul 17 05:27:31.278331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:31.278070    2582 scope.go:117] "RemoveContainer" containerID="a45356ee2281b4aa3092415a3ee765b5438728acc3b952c9b94425c28afafd52"
Jul 17 05:27:31.278331 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:31.278312    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz_odh-ai-gateway-infra(0e7f72ce-5c74-49c3-b7f1-daf5e4547851)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851"
Jul 17 05:27:32.283676 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:32.283654    2582 scope.go:117] "RemoveContainer" containerID="a45356ee2281b4aa3092415a3ee765b5438728acc3b952c9b94425c28afafd52"
Jul 17 05:27:32.284051 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:32.283824    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz_odh-ai-gateway-infra(0e7f72ce-5c74-49c3-b7f1-daf5e4547851)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851"
Jul 17 05:27:34.118710 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.118676    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"]
Jul 17 05:27:34.122893 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.122876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:34.128357 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.128336    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"]
Jul 17 05:27:34.146441 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.146405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afbae7d1-3353-4a18-b900-7469d072d56e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6\" (UID: \"afbae7d1-3353-4a18-b900-7469d072d56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:34.247463 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.247426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afbae7d1-3353-4a18-b900-7469d072d56e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6\" (UID: \"afbae7d1-3353-4a18-b900-7469d072d56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:34.248029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.248009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afbae7d1-3353-4a18-b900-7469d072d56e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6\" (UID: \"afbae7d1-3353-4a18-b900-7469d072d56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:34.434894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.434807    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:34.564920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:34.564798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"]
Jul 17 05:27:34.566924 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:27:34.566892    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbae7d1_3353_4a18_b900_7469d072d56e.slice/crio-5c89456c73eb39d99515d193aafe480e1a5eeec3cc4560a552c3f593b8edf1d9 WatchSource:0}: Error finding container 5c89456c73eb39d99515d193aafe480e1a5eeec3cc4560a552c3f593b8edf1d9: Status 404 returned error can't find the container with id 5c89456c73eb39d99515d193aafe480e1a5eeec3cc4560a552c3f593b8edf1d9
Jul 17 05:27:35.297311 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:35.297277    2582 generic.go:356] "Generic (PLEG): container finished" podID="afbae7d1-3353-4a18-b900-7469d072d56e" containerID="e4293d62f658182065c0f93ffa3e3a1b4d92627693df3fbf65ed2e942ed791da" exitCode=7
Jul 17 05:27:35.297311 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:35.297295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" event={"ID":"afbae7d1-3353-4a18-b900-7469d072d56e","Type":"ContainerDied","Data":"e4293d62f658182065c0f93ffa3e3a1b4d92627693df3fbf65ed2e942ed791da"}
Jul 17 05:27:35.297750 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:35.297352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" event={"ID":"afbae7d1-3353-4a18-b900-7469d072d56e","Type":"ContainerStarted","Data":"5c89456c73eb39d99515d193aafe480e1a5eeec3cc4560a552c3f593b8edf1d9"}
Jul 17 05:27:35.297750 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:35.297663    2582 scope.go:117] "RemoveContainer" containerID="e4293d62f658182065c0f93ffa3e3a1b4d92627693df3fbf65ed2e942ed791da"
Jul 17 05:27:36.303224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:36.303187    2582 generic.go:356] "Generic (PLEG): container finished" podID="afbae7d1-3353-4a18-b900-7469d072d56e" containerID="52b137cc0b2c8e9f1ae63bb4406cf5a8191724a1562223a5bb2df0e17fbd2303" exitCode=7
Jul 17 05:27:36.303689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:36.303241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" event={"ID":"afbae7d1-3353-4a18-b900-7469d072d56e","Type":"ContainerDied","Data":"52b137cc0b2c8e9f1ae63bb4406cf5a8191724a1562223a5bb2df0e17fbd2303"}
Jul 17 05:27:36.303689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:36.303320    2582 scope.go:117] "RemoveContainer" containerID="e4293d62f658182065c0f93ffa3e3a1b4d92627693df3fbf65ed2e942ed791da"
Jul 17 05:27:36.303689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:36.303459    2582 scope.go:117] "RemoveContainer" containerID="52b137cc0b2c8e9f1ae63bb4406cf5a8191724a1562223a5bb2df0e17fbd2303"
Jul 17 05:27:36.303689 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:36.303680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6_odh-ai-gateway-infra(afbae7d1-3353-4a18-b900-7469d072d56e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" podUID="afbae7d1-3353-4a18-b900-7469d072d56e"
Jul 17 05:27:37.309329 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:37.309300    2582 scope.go:117] "RemoveContainer" containerID="52b137cc0b2c8e9f1ae63bb4406cf5a8191724a1562223a5bb2df0e17fbd2303"
Jul 17 05:27:37.309690 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:37.309510    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6_odh-ai-gateway-infra(afbae7d1-3353-4a18-b900-7469d072d56e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" podUID="afbae7d1-3353-4a18-b900-7469d072d56e"
Jul 17 05:27:39.724971 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:39.724936    2582 scope.go:117] "RemoveContainer" containerID="116a0ff87a0bb4fb3dc842a45566fc845f96bd4b68eb1f2563bdca072268cdb3"
Jul 17 05:27:40.322973 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:40.322940    2582 generic.go:356] "Generic (PLEG): container finished" podID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerID="77b0c8d6313f00e38241ce7f3f82b5d325298cf74a38af85e8d4852076ff20d8" exitCode=6
Jul 17 05:27:40.323111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:40.322983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" event={"ID":"73278d02-59dd-4fc8-a2d9-449fce28d880","Type":"ContainerDied","Data":"77b0c8d6313f00e38241ce7f3f82b5d325298cf74a38af85e8d4852076ff20d8"}
Jul 17 05:27:40.323111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:40.323033    2582 scope.go:117] "RemoveContainer" containerID="116a0ff87a0bb4fb3dc842a45566fc845f96bd4b68eb1f2563bdca072268cdb3"
Jul 17 05:27:40.323354 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:40.323320    2582 scope.go:117] "RemoveContainer" containerID="77b0c8d6313f00e38241ce7f3f82b5d325298cf74a38af85e8d4852076ff20d8"
Jul 17 05:27:40.323566 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:40.323545    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-bzmnw_odh-ai-gateway-infra(73278d02-59dd-4fc8-a2d9-449fce28d880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880"
Jul 17 05:27:41.346138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:41.346098    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"]
Jul 17 05:27:41.475734 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:41.475710    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:41.502438 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:41.502412    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73278d02-59dd-4fc8-a2d9-449fce28d880-maas-api-service-ca\") pod \"73278d02-59dd-4fc8-a2d9-449fce28d880\" (UID: \"73278d02-59dd-4fc8-a2d9-449fce28d880\") "
Jul 17 05:27:41.502758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:41.502740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73278d02-59dd-4fc8-a2d9-449fce28d880-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73278d02-59dd-4fc8-a2d9-449fce28d880" (UID: "73278d02-59dd-4fc8-a2d9-449fce28d880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:27:41.603332 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:41.603217    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73278d02-59dd-4fc8-a2d9-449fce28d880-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:27:42.334295 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:42.334198    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"
Jul 17 05:27:42.334295 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:42.334204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw" event={"ID":"73278d02-59dd-4fc8-a2d9-449fce28d880","Type":"ContainerDied","Data":"2c95994be4c3b3efee2ac93ce6a69a385b20eb96d0cb1e0e00fcf1e64def90a4"}
Jul 17 05:27:42.334295 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:42.334286    2582 scope.go:117] "RemoveContainer" containerID="77b0c8d6313f00e38241ce7f3f82b5d325298cf74a38af85e8d4852076ff20d8"
Jul 17 05:27:42.354457 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:42.354424    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"]
Jul 17 05:27:42.360222 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:42.360197    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-bzmnw"]
Jul 17 05:27:42.730132 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:42.730094    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" path="/var/lib/kubelet/pods/73278d02-59dd-4fc8-a2d9-449fce28d880/volumes"
Jul 17 05:27:45.725253 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:45.725222    2582 scope.go:117] "RemoveContainer" containerID="a45356ee2281b4aa3092415a3ee765b5438728acc3b952c9b94425c28afafd52"
Jul 17 05:27:46.354473 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:46.354446    2582 generic.go:356] "Generic (PLEG): container finished" podID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerID="1e628a7921be38fa66e1a925d0ac6fde5a2cde9ac56523a6014c29693ab447a6" exitCode=6
Jul 17 05:27:46.354632 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:46.354494    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" event={"ID":"0e7f72ce-5c74-49c3-b7f1-daf5e4547851","Type":"ContainerDied","Data":"1e628a7921be38fa66e1a925d0ac6fde5a2cde9ac56523a6014c29693ab447a6"}
Jul 17 05:27:46.354632 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:46.354541    2582 scope.go:117] "RemoveContainer" containerID="a45356ee2281b4aa3092415a3ee765b5438728acc3b952c9b94425c28afafd52"
Jul 17 05:27:46.354844 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:46.354814    2582 scope.go:117] "RemoveContainer" containerID="1e628a7921be38fa66e1a925d0ac6fde5a2cde9ac56523a6014c29693ab447a6"
Jul 17 05:27:46.355035 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:46.355015    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz_odh-ai-gateway-infra(0e7f72ce-5c74-49c3-b7f1-daf5e4547851)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851"
Jul 17 05:27:47.380918 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:47.380886    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"]
Jul 17 05:27:47.515446 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:47.515420    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:47.554570 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:47.554533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f72ce-5c74-49c3-b7f1-daf5e4547851-maas-api-service-ca\") pod \"0e7f72ce-5c74-49c3-b7f1-daf5e4547851\" (UID: \"0e7f72ce-5c74-49c3-b7f1-daf5e4547851\") "
Jul 17 05:27:47.554886 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:47.554863    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f72ce-5c74-49c3-b7f1-daf5e4547851-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e7f72ce-5c74-49c3-b7f1-daf5e4547851" (UID: "0e7f72ce-5c74-49c3-b7f1-daf5e4547851"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:27:47.656138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:47.656060    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f72ce-5c74-49c3-b7f1-daf5e4547851-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:27:48.366981 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:48.366946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz" event={"ID":"0e7f72ce-5c74-49c3-b7f1-daf5e4547851","Type":"ContainerDied","Data":"aa2a227173e5cc45ea7290d0a66b8b828a47e3fc0de7eed2610c29bd0593b226"}
Jul 17 05:27:48.366981 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:48.366966    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"
Jul 17 05:27:48.367223 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:48.367001    2582 scope.go:117] "RemoveContainer" containerID="1e628a7921be38fa66e1a925d0ac6fde5a2cde9ac56523a6014c29693ab447a6"
Jul 17 05:27:48.385803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:48.385775    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"]
Jul 17 05:27:48.396607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:48.396579    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-lrfqz"]
Jul 17 05:27:48.729871 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:48.729838    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" path="/var/lib/kubelet/pods/0e7f72ce-5c74-49c3-b7f1-daf5e4547851/volumes"
Jul 17 05:27:51.724762 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:51.724731    2582 scope.go:117] "RemoveContainer" containerID="52b137cc0b2c8e9f1ae63bb4406cf5a8191724a1562223a5bb2df0e17fbd2303"
Jul 17 05:27:52.391490 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:52.391457    2582 generic.go:356] "Generic (PLEG): container finished" podID="afbae7d1-3353-4a18-b900-7469d072d56e" containerID="7eebde7d49bddb23d3875fa3271936101717f089c61ef4d11079f42716ec99d3" exitCode=7
Jul 17 05:27:52.391673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:52.391528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" event={"ID":"afbae7d1-3353-4a18-b900-7469d072d56e","Type":"ContainerDied","Data":"7eebde7d49bddb23d3875fa3271936101717f089c61ef4d11079f42716ec99d3"}
Jul 17 05:27:52.391673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:52.391586    2582 scope.go:117] "RemoveContainer" containerID="52b137cc0b2c8e9f1ae63bb4406cf5a8191724a1562223a5bb2df0e17fbd2303"
Jul 17 05:27:52.391951 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:52.391936    2582 scope.go:117] "RemoveContainer" containerID="7eebde7d49bddb23d3875fa3271936101717f089c61ef4d11079f42716ec99d3"
Jul 17 05:27:52.392170 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:52.392144    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6_odh-ai-gateway-infra(afbae7d1-3353-4a18-b900-7469d072d56e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" podUID="afbae7d1-3353-4a18-b900-7469d072d56e"
Jul 17 05:27:53.415161 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:53.415132    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"]
Jul 17 05:27:53.544814 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:53.544789    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:53.603664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:53.603632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afbae7d1-3353-4a18-b900-7469d072d56e-maas-api-service-ca\") pod \"afbae7d1-3353-4a18-b900-7469d072d56e\" (UID: \"afbae7d1-3353-4a18-b900-7469d072d56e\") "
Jul 17 05:27:53.603988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:53.603962    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbae7d1-3353-4a18-b900-7469d072d56e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afbae7d1-3353-4a18-b900-7469d072d56e" (UID: "afbae7d1-3353-4a18-b900-7469d072d56e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:27:53.705047 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:53.705005    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afbae7d1-3353-4a18-b900-7469d072d56e-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:27:54.402736 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:54.402701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6" event={"ID":"afbae7d1-3353-4a18-b900-7469d072d56e","Type":"ContainerDied","Data":"5c89456c73eb39d99515d193aafe480e1a5eeec3cc4560a552c3f593b8edf1d9"}
Jul 17 05:27:54.402736 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:54.402723    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"
Jul 17 05:27:54.402960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:54.402773    2582 scope.go:117] "RemoveContainer" containerID="7eebde7d49bddb23d3875fa3271936101717f089c61ef4d11079f42716ec99d3"
Jul 17 05:27:54.422448 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:54.422418    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"]
Jul 17 05:27:54.426399 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:54.426365    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lmrv6"]
Jul 17 05:27:54.729962 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:54.729928    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" path="/var/lib/kubelet/pods/afbae7d1-3353-4a18-b900-7469d072d56e/volumes"
Jul 17 05:27:56.819019 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.818968    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"]
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819348    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819376    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819381    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819387    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819393    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819401    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819406    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819416    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819421    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819427    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819433    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819445    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819453    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819506    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819513    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819522    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819532    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819539    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819544    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:27:56.819836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.819550    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:27:56.823730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.823712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:27:56.827362 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.827328    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:27:56.827512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.827480    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:27:56.828483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.828436    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"]
Jul 17 05:27:56.933169 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:56.933125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9213d8-6e9c-4dfa-92bc-638781ce86df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46\" (UID: \"5e9213d8-6e9c-4dfa-92bc-638781ce86df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:27:57.034397 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:57.034355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9213d8-6e9c-4dfa-92bc-638781ce86df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46\" (UID: \"5e9213d8-6e9c-4dfa-92bc-638781ce86df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:27:57.034993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:57.034972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9213d8-6e9c-4dfa-92bc-638781ce86df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46\" (UID: \"5e9213d8-6e9c-4dfa-92bc-638781ce86df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:27:57.135229 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:57.135123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:27:57.271720 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:57.271693    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"]
Jul 17 05:27:57.273892 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:27:57.273864    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9213d8_6e9c_4dfa_92bc_638781ce86df.slice/crio-43860f2e0668b17f882300bf84ebbb132578fc97a88496c18e86aa6e1c0c60b8 WatchSource:0}: Error finding container 43860f2e0668b17f882300bf84ebbb132578fc97a88496c18e86aa6e1c0c60b8: Status 404 returned error can't find the container with id 43860f2e0668b17f882300bf84ebbb132578fc97a88496c18e86aa6e1c0c60b8
Jul 17 05:27:57.418628 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:57.418537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" event={"ID":"5e9213d8-6e9c-4dfa-92bc-638781ce86df","Type":"ContainerStarted","Data":"43860f2e0668b17f882300bf84ebbb132578fc97a88496c18e86aa6e1c0c60b8"}
Jul 17 05:27:58.423922 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:58.423884    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerID="548f89b6ee2395df9a20fb0f216d26a3504b71c93a15d501fa4031c0223a1bb8" exitCode=6
Jul 17 05:27:58.424327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:58.423930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" event={"ID":"5e9213d8-6e9c-4dfa-92bc-638781ce86df","Type":"ContainerDied","Data":"548f89b6ee2395df9a20fb0f216d26a3504b71c93a15d501fa4031c0223a1bb8"}
Jul 17 05:27:58.424327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:58.424152    2582 scope.go:117] "RemoveContainer" containerID="548f89b6ee2395df9a20fb0f216d26a3504b71c93a15d501fa4031c0223a1bb8"
Jul 17 05:27:59.430182 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:59.430100    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerID="d3e598131f1553af3ca1f6250f1757830e85108b634bcb9c89f46926ad19935e" exitCode=6
Jul 17 05:27:59.430182 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:59.430165    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" event={"ID":"5e9213d8-6e9c-4dfa-92bc-638781ce86df","Type":"ContainerDied","Data":"d3e598131f1553af3ca1f6250f1757830e85108b634bcb9c89f46926ad19935e"}
Jul 17 05:27:59.430726 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:59.430226    2582 scope.go:117] "RemoveContainer" containerID="548f89b6ee2395df9a20fb0f216d26a3504b71c93a15d501fa4031c0223a1bb8"
Jul 17 05:27:59.430726 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:27:59.430422    2582 scope.go:117] "RemoveContainer" containerID="d3e598131f1553af3ca1f6250f1757830e85108b634bcb9c89f46926ad19935e"
Jul 17 05:27:59.430726 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:27:59.430633    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46_odh-ai-gateway-infra(5e9213d8-6e9c-4dfa-92bc-638781ce86df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df"
Jul 17 05:28:00.435591 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:00.435560    2582 scope.go:117] "RemoveContainer" containerID="d3e598131f1553af3ca1f6250f1757830e85108b634bcb9c89f46926ad19935e"
Jul 17 05:28:00.436093 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:00.435808    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46_odh-ai-gateway-infra(5e9213d8-6e9c-4dfa-92bc-638781ce86df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df"
Jul 17 05:28:12.725173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:12.725140    2582 scope.go:117] "RemoveContainer" containerID="d3e598131f1553af3ca1f6250f1757830e85108b634bcb9c89f46926ad19935e"
Jul 17 05:28:13.488909 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:13.488879    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerID="e5ef1a0e0e91a98c4fa84ba185008cb4ec32c35b5397834b97129a759da164bb" exitCode=6
Jul 17 05:28:13.489082 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:13.488962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" event={"ID":"5e9213d8-6e9c-4dfa-92bc-638781ce86df","Type":"ContainerDied","Data":"e5ef1a0e0e91a98c4fa84ba185008cb4ec32c35b5397834b97129a759da164bb"}
Jul 17 05:28:13.489082 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:13.489030    2582 scope.go:117] "RemoveContainer" containerID="d3e598131f1553af3ca1f6250f1757830e85108b634bcb9c89f46926ad19935e"
Jul 17 05:28:13.489371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:13.489351    2582 scope.go:117] "RemoveContainer" containerID="e5ef1a0e0e91a98c4fa84ba185008cb4ec32c35b5397834b97129a759da164bb"
Jul 17 05:28:13.489575 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:13.489554    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46_odh-ai-gateway-infra(5e9213d8-6e9c-4dfa-92bc-638781ce86df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df"
Jul 17 05:28:14.516153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:14.516117    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"]
Jul 17 05:28:14.648989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:14.648966    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:28:14.784935 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:14.784851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9213d8-6e9c-4dfa-92bc-638781ce86df-maas-api-service-ca\") pod \"5e9213d8-6e9c-4dfa-92bc-638781ce86df\" (UID: \"5e9213d8-6e9c-4dfa-92bc-638781ce86df\") "
Jul 17 05:28:14.785210 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:14.785189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9213d8-6e9c-4dfa-92bc-638781ce86df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e9213d8-6e9c-4dfa-92bc-638781ce86df" (UID: "5e9213d8-6e9c-4dfa-92bc-638781ce86df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:28:14.886409 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:14.886373    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9213d8-6e9c-4dfa-92bc-638781ce86df-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:28:15.500561 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:15.500532    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"
Jul 17 05:28:15.500561 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:15.500559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46" event={"ID":"5e9213d8-6e9c-4dfa-92bc-638781ce86df","Type":"ContainerDied","Data":"43860f2e0668b17f882300bf84ebbb132578fc97a88496c18e86aa6e1c0c60b8"}
Jul 17 05:28:15.500828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:15.500620    2582 scope.go:117] "RemoveContainer" containerID="e5ef1a0e0e91a98c4fa84ba185008cb4ec32c35b5397834b97129a759da164bb"
Jul 17 05:28:15.523649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:15.523625    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"]
Jul 17 05:28:15.530604 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:15.530578    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-g8j46"]
Jul 17 05:28:16.730110 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:16.730080    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" path="/var/lib/kubelet/pods/5e9213d8-6e9c-4dfa-92bc-638781ce86df/volumes"
Jul 17 05:28:18.758500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758461    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"]
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758768    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758781    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758797    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758802    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="afbae7d1-3353-4a18-b900-7469d072d56e" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758810    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758815    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758827    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758838    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758884    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758891    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7f72ce-5c74-49c3-b7f1-daf5e4547851" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758898    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73278d02-59dd-4fc8-a2d9-449fce28d880" containerName="revoke-keys"
Jul 17 05:28:18.759023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.758908    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:18.763331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.763311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:18.766402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.766379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:28:18.766543 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.766441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:28:18.769841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.769791    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"]
Jul 17 05:28:18.917362 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:18.917322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd\" (UID: \"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:19.018062 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.017983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd\" (UID: \"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:19.018645 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.018622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd\" (UID: \"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:19.074558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.074508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:19.207871 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.207837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"]
Jul 17 05:28:19.210885 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:28:19.210851    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b50cedd_5aa9_4a19_a4cd_8b02cae8642b.slice/crio-7fd1a574d7fbbdf7f94e4c8f3c06c6a35c139756d6f2e247ab0f5678bb16ff8b WatchSource:0}: Error finding container 7fd1a574d7fbbdf7f94e4c8f3c06c6a35c139756d6f2e247ab0f5678bb16ff8b: Status 404 returned error can't find the container with id 7fd1a574d7fbbdf7f94e4c8f3c06c6a35c139756d6f2e247ab0f5678bb16ff8b
Jul 17 05:28:19.521512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.521480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" event={"ID":"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b","Type":"ContainerStarted","Data":"846ba3a986b4a29260cdba62f849dfe96bec1e4a21c82bc55678a3a705bf0674"}
Jul 17 05:28:19.521705 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.521535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" event={"ID":"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b","Type":"ContainerStarted","Data":"7fd1a574d7fbbdf7f94e4c8f3c06c6a35c139756d6f2e247ab0f5678bb16ff8b"}
Jul 17 05:28:19.538018 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:19.537971    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" podStartSLOduration=1.53795543 podStartE2EDuration="1.53795543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:28:19.53662572 +0000 UTC m=+2303.273688080" watchObservedRunningTime="2026-07-17 05:28:19.53795543 +0000 UTC m=+2303.275017796"
Jul 17 05:28:20.527101 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.527073    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerID="846ba3a986b4a29260cdba62f849dfe96bec1e4a21c82bc55678a3a705bf0674" exitCode=6
Jul 17 05:28:20.527530 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.527148    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" event={"ID":"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b","Type":"ContainerDied","Data":"846ba3a986b4a29260cdba62f849dfe96bec1e4a21c82bc55678a3a705bf0674"}
Jul 17 05:28:20.527530 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.527343    2582 scope.go:117] "RemoveContainer" containerID="846ba3a986b4a29260cdba62f849dfe96bec1e4a21c82bc55678a3a705bf0674"
Jul 17 05:28:20.629175 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.629140    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"]
Jul 17 05:28:20.629472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.629459    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:20.629520 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.629474    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:20.629555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.629521    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9213d8-6e9c-4dfa-92bc-638781ce86df" containerName="revoke-keys"
Jul 17 05:28:20.633688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.633671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:20.639129 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.639098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"]
Jul 17 05:28:20.832247 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.832148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5248003-df34-4a80-aad8-360b907a7f21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-5rgjj\" (UID: \"f5248003-df34-4a80-aad8-360b907a7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:20.933022 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.932980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5248003-df34-4a80-aad8-360b907a7f21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-5rgjj\" (UID: \"f5248003-df34-4a80-aad8-360b907a7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:20.933654 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.933635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5248003-df34-4a80-aad8-360b907a7f21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-5rgjj\" (UID: \"f5248003-df34-4a80-aad8-360b907a7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:20.945046 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:20.945020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:21.077087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.077064    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"]
Jul 17 05:28:21.078202 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:28:21.078177    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5248003_df34_4a80_aad8_360b907a7f21.slice/crio-bd3fa59a09c2958abccf799570fd7a2d007a971ec446d3c70794c410ff8ba184 WatchSource:0}: Error finding container bd3fa59a09c2958abccf799570fd7a2d007a971ec446d3c70794c410ff8ba184: Status 404 returned error can't find the container with id bd3fa59a09c2958abccf799570fd7a2d007a971ec446d3c70794c410ff8ba184
Jul 17 05:28:21.532542 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.532506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" event={"ID":"f5248003-df34-4a80-aad8-360b907a7f21","Type":"ContainerStarted","Data":"d1fd0916a13b9b9f4136194fbb6515ea1ae1119df15a8936b50d9dba314269e6"}
Jul 17 05:28:21.533008 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.532565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" event={"ID":"f5248003-df34-4a80-aad8-360b907a7f21","Type":"ContainerStarted","Data":"bd3fa59a09c2958abccf799570fd7a2d007a971ec446d3c70794c410ff8ba184"}
Jul 17 05:28:21.534057 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.534032    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerID="a64e2bcc8d2c1532c135d0934296b4e2e79b76b3220336787918160f4e19af25" exitCode=6
Jul 17 05:28:21.534127 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.534099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" event={"ID":"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b","Type":"ContainerDied","Data":"a64e2bcc8d2c1532c135d0934296b4e2e79b76b3220336787918160f4e19af25"}
Jul 17 05:28:21.534164 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.534146    2582 scope.go:117] "RemoveContainer" containerID="846ba3a986b4a29260cdba62f849dfe96bec1e4a21c82bc55678a3a705bf0674"
Jul 17 05:28:21.534327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.534312    2582 scope.go:117] "RemoveContainer" containerID="a64e2bcc8d2c1532c135d0934296b4e2e79b76b3220336787918160f4e19af25"
Jul 17 05:28:21.534530 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:21.534512    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd_odh-ai-gateway-infra(4b50cedd-5aa9-4a19-a4cd-8b02cae8642b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b"
Jul 17 05:28:21.547128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:21.547085    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" podStartSLOduration=1.547071062 podStartE2EDuration="1.547071062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:28:21.546613827 +0000 UTC m=+2305.283676155" watchObservedRunningTime="2026-07-17 05:28:21.547071062 +0000 UTC m=+2305.284133397"
Jul 17 05:28:22.539607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:22.539576    2582 scope.go:117] "RemoveContainer" containerID="a64e2bcc8d2c1532c135d0934296b4e2e79b76b3220336787918160f4e19af25"
Jul 17 05:28:22.540063 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:22.539811    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd_odh-ai-gateway-infra(4b50cedd-5aa9-4a19-a4cd-8b02cae8642b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b"
Jul 17 05:28:22.540873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:22.540849    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5248003-df34-4a80-aad8-360b907a7f21" containerID="d1fd0916a13b9b9f4136194fbb6515ea1ae1119df15a8936b50d9dba314269e6" exitCode=6
Jul 17 05:28:22.540998 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:22.540894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" event={"ID":"f5248003-df34-4a80-aad8-360b907a7f21","Type":"ContainerDied","Data":"d1fd0916a13b9b9f4136194fbb6515ea1ae1119df15a8936b50d9dba314269e6"}
Jul 17 05:28:22.541097 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:22.541083    2582 scope.go:117] "RemoveContainer" containerID="d1fd0916a13b9b9f4136194fbb6515ea1ae1119df15a8936b50d9dba314269e6"
Jul 17 05:28:23.553560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:23.553515    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5248003-df34-4a80-aad8-360b907a7f21" containerID="ac7f0aa5b2596c8c39d423cebeb6e27b34e0dad4f698daa1eb32a9646660b0e2" exitCode=6
Jul 17 05:28:23.554037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:23.553594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" event={"ID":"f5248003-df34-4a80-aad8-360b907a7f21","Type":"ContainerDied","Data":"ac7f0aa5b2596c8c39d423cebeb6e27b34e0dad4f698daa1eb32a9646660b0e2"}
Jul 17 05:28:23.554037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:23.553684    2582 scope.go:117] "RemoveContainer" containerID="d1fd0916a13b9b9f4136194fbb6515ea1ae1119df15a8936b50d9dba314269e6"
Jul 17 05:28:23.554037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:23.553854    2582 scope.go:117] "RemoveContainer" containerID="ac7f0aa5b2596c8c39d423cebeb6e27b34e0dad4f698daa1eb32a9646660b0e2"
Jul 17 05:28:23.554200 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:23.554073    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-5rgjj_odh-ai-gateway-infra(f5248003-df34-4a80-aad8-360b907a7f21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" podUID="f5248003-df34-4a80-aad8-360b907a7f21"
Jul 17 05:28:24.559529 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:24.559504    2582 scope.go:117] "RemoveContainer" containerID="ac7f0aa5b2596c8c39d423cebeb6e27b34e0dad4f698daa1eb32a9646660b0e2"
Jul 17 05:28:24.559903 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:24.559698    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-5rgjj_odh-ai-gateway-infra(f5248003-df34-4a80-aad8-360b907a7f21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" podUID="f5248003-df34-4a80-aad8-360b907a7f21"
Jul 17 05:28:33.725422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:33.725385    2582 scope.go:117] "RemoveContainer" containerID="a64e2bcc8d2c1532c135d0934296b4e2e79b76b3220336787918160f4e19af25"
Jul 17 05:28:34.178193 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.178154    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"]
Jul 17 05:28:34.181288 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.181248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:34.190027 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.189998    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"]
Jul 17 05:28:34.233416 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.233380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3250b57-eb55-4b99-9ae4-691768d1db27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr\" (UID: \"e3250b57-eb55-4b99-9ae4-691768d1db27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:34.334689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.334652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3250b57-eb55-4b99-9ae4-691768d1db27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr\" (UID: \"e3250b57-eb55-4b99-9ae4-691768d1db27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:34.335234 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.335216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3250b57-eb55-4b99-9ae4-691768d1db27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr\" (UID: \"e3250b57-eb55-4b99-9ae4-691768d1db27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:34.492025 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.491966    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:34.602370 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.602337    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerID="5424e3d061f357e5ba24a51e3778e8b786d9c5d2c354430b2a5d2c566e697afe" exitCode=6
Jul 17 05:28:34.602520 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.602412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" event={"ID":"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b","Type":"ContainerDied","Data":"5424e3d061f357e5ba24a51e3778e8b786d9c5d2c354430b2a5d2c566e697afe"}
Jul 17 05:28:34.602520 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.602487    2582 scope.go:117] "RemoveContainer" containerID="a64e2bcc8d2c1532c135d0934296b4e2e79b76b3220336787918160f4e19af25"
Jul 17 05:28:34.602843 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.602818    2582 scope.go:117] "RemoveContainer" containerID="5424e3d061f357e5ba24a51e3778e8b786d9c5d2c354430b2a5d2c566e697afe"
Jul 17 05:28:34.603089 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:34.603067    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd_odh-ai-gateway-infra(4b50cedd-5aa9-4a19-a4cd-8b02cae8642b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b"
Jul 17 05:28:34.624327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:34.624302    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"]
Jul 17 05:28:34.626875 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:28:34.626851    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3250b57_eb55_4b99_9ae4_691768d1db27.slice/crio-85a2f6388555c001d83651f8e8c325033705e557826240d9f029976748d0a261 WatchSource:0}: Error finding container 85a2f6388555c001d83651f8e8c325033705e557826240d9f029976748d0a261: Status 404 returned error can't find the container with id 85a2f6388555c001d83651f8e8c325033705e557826240d9f029976748d0a261
Jul 17 05:28:35.608789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.608694    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerID="e6e2c75113e69c001e9e7a1d1ea6ebf2a45ee38d580bd544108fc9d9adc51113" exitCode=7
Jul 17 05:28:35.608789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.608778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" event={"ID":"e3250b57-eb55-4b99-9ae4-691768d1db27","Type":"ContainerDied","Data":"e6e2c75113e69c001e9e7a1d1ea6ebf2a45ee38d580bd544108fc9d9adc51113"}
Jul 17 05:28:35.609316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.608830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" event={"ID":"e3250b57-eb55-4b99-9ae4-691768d1db27","Type":"ContainerStarted","Data":"85a2f6388555c001d83651f8e8c325033705e557826240d9f029976748d0a261"}
Jul 17 05:28:35.609316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.608990    2582 scope.go:117] "RemoveContainer" containerID="e6e2c75113e69c001e9e7a1d1ea6ebf2a45ee38d580bd544108fc9d9adc51113"
Jul 17 05:28:35.631113 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.631087    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"]
Jul 17 05:28:35.760147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.760123    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:35.848346 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.848307    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b-maas-api-service-ca\") pod \"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b\" (UID: \"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b\") "
Jul 17 05:28:35.848685 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.848662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" (UID: "4b50cedd-5aa9-4a19-a4cd-8b02cae8642b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:28:35.949225 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:35.949183    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:28:36.619081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.619053    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"
Jul 17 05:28:36.619570 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.619092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd" event={"ID":"4b50cedd-5aa9-4a19-a4cd-8b02cae8642b","Type":"ContainerDied","Data":"7fd1a574d7fbbdf7f94e4c8f3c06c6a35c139756d6f2e247ab0f5678bb16ff8b"}
Jul 17 05:28:36.619570 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.619188    2582 scope.go:117] "RemoveContainer" containerID="5424e3d061f357e5ba24a51e3778e8b786d9c5d2c354430b2a5d2c566e697afe"
Jul 17 05:28:36.621149 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.621121    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerID="f5a7193842d27635fc09b8364e63f1f11ad77568d58e54aacb1daa202d786b43" exitCode=7
Jul 17 05:28:36.621371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.621170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" event={"ID":"e3250b57-eb55-4b99-9ae4-691768d1db27","Type":"ContainerDied","Data":"f5a7193842d27635fc09b8364e63f1f11ad77568d58e54aacb1daa202d786b43"}
Jul 17 05:28:36.621447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.621397    2582 scope.go:117] "RemoveContainer" containerID="f5a7193842d27635fc09b8364e63f1f11ad77568d58e54aacb1daa202d786b43"
Jul 17 05:28:36.621601 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:36.621579    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr_odh-ai-gateway-infra(e3250b57-eb55-4b99-9ae4-691768d1db27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27"
Jul 17 05:28:36.627614 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.627597    2582 scope.go:117] "RemoveContainer" containerID="e6e2c75113e69c001e9e7a1d1ea6ebf2a45ee38d580bd544108fc9d9adc51113"
Jul 17 05:28:36.657466 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.657432    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"]
Jul 17 05:28:36.661679 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.661655    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-xbrzd"]
Jul 17 05:28:36.729683 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:36.729652    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" path="/var/lib/kubelet/pods/4b50cedd-5aa9-4a19-a4cd-8b02cae8642b/volumes"
Jul 17 05:28:37.628413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:37.628388    2582 scope.go:117] "RemoveContainer" containerID="f5a7193842d27635fc09b8364e63f1f11ad77568d58e54aacb1daa202d786b43"
Jul 17 05:28:37.628897 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:37.628643    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr_odh-ai-gateway-infra(e3250b57-eb55-4b99-9ae4-691768d1db27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27"
Jul 17 05:28:38.725417 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:38.725388    2582 scope.go:117] "RemoveContainer" containerID="ac7f0aa5b2596c8c39d423cebeb6e27b34e0dad4f698daa1eb32a9646660b0e2"
Jul 17 05:28:39.637590 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:39.637502    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5248003-df34-4a80-aad8-360b907a7f21" containerID="6434665cb4d9a66f0dc66662f670e5c9a07bbf4fdda964981a586b950a322fa6" exitCode=6
Jul 17 05:28:39.637590 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:39.637577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" event={"ID":"f5248003-df34-4a80-aad8-360b907a7f21","Type":"ContainerDied","Data":"6434665cb4d9a66f0dc66662f670e5c9a07bbf4fdda964981a586b950a322fa6"}
Jul 17 05:28:39.637804 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:39.637636    2582 scope.go:117] "RemoveContainer" containerID="ac7f0aa5b2596c8c39d423cebeb6e27b34e0dad4f698daa1eb32a9646660b0e2"
Jul 17 05:28:39.637893 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:39.637878    2582 scope.go:117] "RemoveContainer" containerID="6434665cb4d9a66f0dc66662f670e5c9a07bbf4fdda964981a586b950a322fa6"
Jul 17 05:28:39.638073 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:39.638056    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-5rgjj_odh-ai-gateway-infra(f5248003-df34-4a80-aad8-360b907a7f21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" podUID="f5248003-df34-4a80-aad8-360b907a7f21"
Jul 17 05:28:40.661488 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:40.661458    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"]
Jul 17 05:28:40.795770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:40.795741    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:40.894794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:40.894755    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5248003-df34-4a80-aad8-360b907a7f21-maas-api-service-ca\") pod \"f5248003-df34-4a80-aad8-360b907a7f21\" (UID: \"f5248003-df34-4a80-aad8-360b907a7f21\") "
Jul 17 05:28:40.895094 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:40.895072    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5248003-df34-4a80-aad8-360b907a7f21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5248003-df34-4a80-aad8-360b907a7f21" (UID: "f5248003-df34-4a80-aad8-360b907a7f21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:28:40.995682 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:40.995646    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5248003-df34-4a80-aad8-360b907a7f21-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:28:41.648930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:41.648901    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"
Jul 17 05:28:41.648930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:41.648911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj" event={"ID":"f5248003-df34-4a80-aad8-360b907a7f21","Type":"ContainerDied","Data":"bd3fa59a09c2958abccf799570fd7a2d007a971ec446d3c70794c410ff8ba184"}
Jul 17 05:28:41.649201 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:41.648970    2582 scope.go:117] "RemoveContainer" containerID="6434665cb4d9a66f0dc66662f670e5c9a07bbf4fdda964981a586b950a322fa6"
Jul 17 05:28:41.676245 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:41.676212    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"]
Jul 17 05:28:41.682296 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:41.682248    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-5rgjj"]
Jul 17 05:28:42.729611 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:42.729577    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5248003-df34-4a80-aad8-360b907a7f21" path="/var/lib/kubelet/pods/f5248003-df34-4a80-aad8-360b907a7f21/volumes"
Jul 17 05:28:46.862689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"]
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862890    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862901    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862916    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862921    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862930    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862935    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862963    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.862968    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.863017    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.863027    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.863033    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:28:46.863099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.863041    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:28:46.867094 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.867076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:28:46.873554 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.873529    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"]
Jul 17 05:28:46.945551 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:46.945516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d\" (UID: \"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:28:47.046835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.046796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d\" (UID: \"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:28:47.047437 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.047417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d\" (UID: \"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:28:47.179521 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.179433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:28:47.328425 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.328376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"]
Jul 17 05:28:47.331337 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:28:47.331309    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcf02d9_2df8_47fe_afb1_aebf174a3c0c.slice/crio-ab57c5b4664208a0ca30c4de8fe77befedabd486c5add3091c285c290a8b1813 WatchSource:0}: Error finding container ab57c5b4664208a0ca30c4de8fe77befedabd486c5add3091c285c290a8b1813: Status 404 returned error can't find the container with id ab57c5b4664208a0ca30c4de8fe77befedabd486c5add3091c285c290a8b1813
Jul 17 05:28:47.675715 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.675677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" event={"ID":"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c","Type":"ContainerStarted","Data":"516a559df9251a59fd868716578c30960dcb6796873ef20b2c00fe9192c67419"}
Jul 17 05:28:47.675898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.675736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" event={"ID":"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c","Type":"ContainerStarted","Data":"ab57c5b4664208a0ca30c4de8fe77befedabd486c5add3091c285c290a8b1813"}
Jul 17 05:28:47.692638 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:47.692587    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" podStartSLOduration=1.692571079 podStartE2EDuration="1.692571079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:28:47.692029608 +0000 UTC m=+2331.429091945" watchObservedRunningTime="2026-07-17 05:28:47.692571079 +0000 UTC m=+2331.429633414"
Jul 17 05:28:48.681621 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:48.681585    2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerID="516a559df9251a59fd868716578c30960dcb6796873ef20b2c00fe9192c67419" exitCode=6
Jul 17 05:28:48.682005 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:48.681670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" event={"ID":"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c","Type":"ContainerDied","Data":"516a559df9251a59fd868716578c30960dcb6796873ef20b2c00fe9192c67419"}
Jul 17 05:28:48.682005 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:48.681822    2582 scope.go:117] "RemoveContainer" containerID="516a559df9251a59fd868716578c30960dcb6796873ef20b2c00fe9192c67419"
Jul 17 05:28:49.688309 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:49.688198    2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerID="26fb3cc9fadb65735015ba8d3e35e6f84ad028c87674401c2db0214b791022a0" exitCode=6
Jul 17 05:28:49.688867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:49.688307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" event={"ID":"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c","Type":"ContainerDied","Data":"26fb3cc9fadb65735015ba8d3e35e6f84ad028c87674401c2db0214b791022a0"}
Jul 17 05:28:49.688867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:49.688369    2582 scope.go:117] "RemoveContainer" containerID="516a559df9251a59fd868716578c30960dcb6796873ef20b2c00fe9192c67419"
Jul 17 05:28:49.688867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:49.688584    2582 scope.go:117] "RemoveContainer" containerID="26fb3cc9fadb65735015ba8d3e35e6f84ad028c87674401c2db0214b791022a0"
Jul 17 05:28:49.688867 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:49.688793    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d_odh-ai-gateway-infra(9fcf02d9-2df8-47fe-afb1-aebf174a3c0c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c"
Jul 17 05:28:50.694153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:50.694121    2582 scope.go:117] "RemoveContainer" containerID="26fb3cc9fadb65735015ba8d3e35e6f84ad028c87674401c2db0214b791022a0"
Jul 17 05:28:50.694654 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:50.694319    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d_odh-ai-gateway-infra(9fcf02d9-2df8-47fe-afb1-aebf174a3c0c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c"
Jul 17 05:28:50.725095 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:50.725070    2582 scope.go:117] "RemoveContainer" containerID="f5a7193842d27635fc09b8364e63f1f11ad77568d58e54aacb1daa202d786b43"
Jul 17 05:28:51.700154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:51.700121    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerID="9fd5e66bd29e8c99d47dd3d3eb68010c7bf7cf1389fbd545c5c5a060564d88c0" exitCode=7
Jul 17 05:28:51.700656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:51.700175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" event={"ID":"e3250b57-eb55-4b99-9ae4-691768d1db27","Type":"ContainerDied","Data":"9fd5e66bd29e8c99d47dd3d3eb68010c7bf7cf1389fbd545c5c5a060564d88c0"}
Jul 17 05:28:51.700656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:51.700222    2582 scope.go:117] "RemoveContainer" containerID="f5a7193842d27635fc09b8364e63f1f11ad77568d58e54aacb1daa202d786b43"
Jul 17 05:28:51.700656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:51.700573    2582 scope.go:117] "RemoveContainer" containerID="9fd5e66bd29e8c99d47dd3d3eb68010c7bf7cf1389fbd545c5c5a060564d88c0"
Jul 17 05:28:51.700858 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:28:51.700835    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr_odh-ai-gateway-infra(e3250b57-eb55-4b99-9ae4-691768d1db27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27"
Jul 17 05:28:52.730595 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:52.730555    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"]
Jul 17 05:28:52.869279 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:52.869228    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:52.997228 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:52.997137    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3250b57-eb55-4b99-9ae4-691768d1db27-maas-api-service-ca\") pod \"e3250b57-eb55-4b99-9ae4-691768d1db27\" (UID: \"e3250b57-eb55-4b99-9ae4-691768d1db27\") "
Jul 17 05:28:52.997515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:52.997493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3250b57-eb55-4b99-9ae4-691768d1db27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3250b57-eb55-4b99-9ae4-691768d1db27" (UID: "e3250b57-eb55-4b99-9ae4-691768d1db27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:28:53.097968 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:53.097928    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3250b57-eb55-4b99-9ae4-691768d1db27-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:28:53.717490 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:53.717450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr" event={"ID":"e3250b57-eb55-4b99-9ae4-691768d1db27","Type":"ContainerDied","Data":"85a2f6388555c001d83651f8e8c325033705e557826240d9f029976748d0a261"}
Jul 17 05:28:53.717490 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:53.717477    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"
Jul 17 05:28:53.717727 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:53.717554    2582 scope.go:117] "RemoveContainer" containerID="9fd5e66bd29e8c99d47dd3d3eb68010c7bf7cf1389fbd545c5c5a060564d88c0"
Jul 17 05:28:53.741168 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:53.741138    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"]
Jul 17 05:28:53.748355 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:53.748330    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ps7nr"]
Jul 17 05:28:54.729789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:28:54.729752    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" path="/var/lib/kubelet/pods/e3250b57-eb55-4b99-9ae4-691768d1db27/volumes"
Jul 17 05:29:04.725359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:04.725325    2582 scope.go:117] "RemoveContainer" containerID="26fb3cc9fadb65735015ba8d3e35e6f84ad028c87674401c2db0214b791022a0"
Jul 17 05:29:05.770453 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:05.770415    2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerID="eaa95426ac3417fd50a7c10d6cb64b2e09efbd0c64d2f89d82280a1b8c9e9eb6" exitCode=6
Jul 17 05:29:05.770902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:05.770867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" event={"ID":"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c","Type":"ContainerDied","Data":"eaa95426ac3417fd50a7c10d6cb64b2e09efbd0c64d2f89d82280a1b8c9e9eb6"}
Jul 17 05:29:05.771000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:05.770984    2582 scope.go:117] "RemoveContainer" containerID="26fb3cc9fadb65735015ba8d3e35e6f84ad028c87674401c2db0214b791022a0"
Jul 17 05:29:05.771531 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:05.771512    2582 scope.go:117] "RemoveContainer" containerID="eaa95426ac3417fd50a7c10d6cb64b2e09efbd0c64d2f89d82280a1b8c9e9eb6"
Jul 17 05:29:05.771929 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:05.771890    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d_odh-ai-gateway-infra(9fcf02d9-2df8-47fe-afb1-aebf174a3c0c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c"
Jul 17 05:29:06.799392 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:06.799357    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"]
Jul 17 05:29:06.929242 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:06.929218    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:29:07.020614 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.020585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c-maas-api-service-ca\") pod \"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c\" (UID: \"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c\") "
Jul 17 05:29:07.020953 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.020932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" (UID: "9fcf02d9-2df8-47fe-afb1-aebf174a3c0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:29:07.122029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.121935    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:29:07.781305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.781270    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"
Jul 17 05:29:07.781485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.781273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d" event={"ID":"9fcf02d9-2df8-47fe-afb1-aebf174a3c0c","Type":"ContainerDied","Data":"ab57c5b4664208a0ca30c4de8fe77befedabd486c5add3091c285c290a8b1813"}
Jul 17 05:29:07.781485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.781406    2582 scope.go:117] "RemoveContainer" containerID="eaa95426ac3417fd50a7c10d6cb64b2e09efbd0c64d2f89d82280a1b8c9e9eb6"
Jul 17 05:29:07.802221 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.802194    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"]
Jul 17 05:29:07.804206 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:07.804183    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-wkb7d"]
Jul 17 05:29:08.729884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.729845    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" path="/var/lib/kubelet/pods/9fcf02d9-2df8-47fe-afb1-aebf174a3c0c/volumes"
Jul 17 05:29:08.814567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814536    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"]
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814802    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814812    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814820    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814825    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814833    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814839    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814849    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814854    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814863    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814868    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814876    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814881    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814927    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814934    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:08.814932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814941    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b50cedd-5aa9-4a19-a4cd-8b02cae8642b" containerName="revoke-keys"
Jul 17 05:29:08.815422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814947    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5248003-df34-4a80-aad8-360b907a7f21" containerName="revoke-keys"
Jul 17 05:29:08.815422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814953    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:08.815422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.814961    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:08.818909 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.818892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:08.822151 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.822133    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:29:08.822429 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.822410    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:29:08.841771 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.841747    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"]
Jul 17 05:29:08.936121 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:08.936083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc8772c-ecbd-46ab-bc7e-d53c70e14867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp\" (UID: \"bdc8772c-ecbd-46ab-bc7e-d53c70e14867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:09.037143 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.037048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc8772c-ecbd-46ab-bc7e-d53c70e14867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp\" (UID: \"bdc8772c-ecbd-46ab-bc7e-d53c70e14867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:09.037717 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.037691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc8772c-ecbd-46ab-bc7e-d53c70e14867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp\" (UID: \"bdc8772c-ecbd-46ab-bc7e-d53c70e14867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:09.129077 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.129034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:09.277771 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.277734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"]
Jul 17 05:29:09.292034 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:29:09.292000    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc8772c_ecbd_46ab_bc7e_d53c70e14867.slice/crio-b756a39cea1189e8eb05316fb29dc946aaf2823ad062a94556af23e5e043305b WatchSource:0}: Error finding container b756a39cea1189e8eb05316fb29dc946aaf2823ad062a94556af23e5e043305b: Status 404 returned error can't find the container with id b756a39cea1189e8eb05316fb29dc946aaf2823ad062a94556af23e5e043305b
Jul 17 05:29:09.792602 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.792575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" event={"ID":"bdc8772c-ecbd-46ab-bc7e-d53c70e14867","Type":"ContainerStarted","Data":"bc53d1e2daf420ff9e80300ce75c5fa22bd9ef6a1b493d760d136e47dd864e10"}
Jul 17 05:29:09.792728 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.792624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" event={"ID":"bdc8772c-ecbd-46ab-bc7e-d53c70e14867","Type":"ContainerStarted","Data":"b756a39cea1189e8eb05316fb29dc946aaf2823ad062a94556af23e5e043305b"}
Jul 17 05:29:09.810871 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:09.810825    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" podStartSLOduration=1.8108083430000002 podStartE2EDuration="1.810808343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:29:09.809696988 +0000 UTC m=+2353.546759324" watchObservedRunningTime="2026-07-17 05:29:09.810808343 +0000 UTC m=+2353.547870678"
Jul 17 05:29:10.800692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:10.800660    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerID="bc53d1e2daf420ff9e80300ce75c5fa22bd9ef6a1b493d760d136e47dd864e10" exitCode=6
Jul 17 05:29:10.801079 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:10.800723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" event={"ID":"bdc8772c-ecbd-46ab-bc7e-d53c70e14867","Type":"ContainerDied","Data":"bc53d1e2daf420ff9e80300ce75c5fa22bd9ef6a1b493d760d136e47dd864e10"}
Jul 17 05:29:10.801079 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:10.800988    2582 scope.go:117] "RemoveContainer" containerID="bc53d1e2daf420ff9e80300ce75c5fa22bd9ef6a1b493d760d136e47dd864e10"
Jul 17 05:29:11.806692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:11.806659    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerID="f09666a9fb1f1ec6887d35df85628aea72ec7bccc74e26ed4c8c4a3e000721f4" exitCode=6
Jul 17 05:29:11.807081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:11.806751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" event={"ID":"bdc8772c-ecbd-46ab-bc7e-d53c70e14867","Type":"ContainerDied","Data":"f09666a9fb1f1ec6887d35df85628aea72ec7bccc74e26ed4c8c4a3e000721f4"}
Jul 17 05:29:11.807081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:11.806803    2582 scope.go:117] "RemoveContainer" containerID="bc53d1e2daf420ff9e80300ce75c5fa22bd9ef6a1b493d760d136e47dd864e10"
Jul 17 05:29:11.807081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:11.806966    2582 scope.go:117] "RemoveContainer" containerID="f09666a9fb1f1ec6887d35df85628aea72ec7bccc74e26ed4c8c4a3e000721f4"
Jul 17 05:29:11.807212 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:11.807182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp_odh-ai-gateway-infra(bdc8772c-ecbd-46ab-bc7e-d53c70e14867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867"
Jul 17 05:29:12.812091 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:12.812058    2582 scope.go:117] "RemoveContainer" containerID="f09666a9fb1f1ec6887d35df85628aea72ec7bccc74e26ed4c8c4a3e000721f4"
Jul 17 05:29:12.812525 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:12.812302    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp_odh-ai-gateway-infra(bdc8772c-ecbd-46ab-bc7e-d53c70e14867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867"
Jul 17 05:29:20.675253 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675218    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"]
Jul 17 05:29:20.675693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675566    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:20.675693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675580    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:20.675693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675597    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:20.675693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675602    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:20.675693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675649    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3250b57-eb55-4b99-9ae4-691768d1db27" containerName="revoke-keys"
Jul 17 05:29:20.675693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.675658    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcf02d9-2df8-47fe-afb1-aebf174a3c0c" containerName="revoke-keys"
Jul 17 05:29:20.679803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.679785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:20.687312 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.687285    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"]
Jul 17 05:29:20.727800 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.727762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d425ad-1abd-4244-83cd-8fee4eaf68e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6c78q\" (UID: \"b0d425ad-1abd-4244-83cd-8fee4eaf68e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:20.828623 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.828583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d425ad-1abd-4244-83cd-8fee4eaf68e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6c78q\" (UID: \"b0d425ad-1abd-4244-83cd-8fee4eaf68e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:20.829245 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.829218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d425ad-1abd-4244-83cd-8fee4eaf68e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6c78q\" (UID: \"b0d425ad-1abd-4244-83cd-8fee4eaf68e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:20.991743 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:20.991706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:21.121916 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:21.121886    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"]
Jul 17 05:29:21.124715 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:29:21.124689    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d425ad_1abd_4244_83cd_8fee4eaf68e6.slice/crio-4f57d4fbe7413a84d8a2db853eb2bc10e403e147e759115aa9c834a415eb2bff WatchSource:0}: Error finding container 4f57d4fbe7413a84d8a2db853eb2bc10e403e147e759115aa9c834a415eb2bff: Status 404 returned error can't find the container with id 4f57d4fbe7413a84d8a2db853eb2bc10e403e147e759115aa9c834a415eb2bff
Jul 17 05:29:21.848366 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:21.848331    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerID="3773967144b34b6a776280086a6b26f3c8e6f500b382fff384cc728acbeaf3f4" exitCode=6
Jul 17 05:29:21.848808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:21.848399    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" event={"ID":"b0d425ad-1abd-4244-83cd-8fee4eaf68e6","Type":"ContainerDied","Data":"3773967144b34b6a776280086a6b26f3c8e6f500b382fff384cc728acbeaf3f4"}
Jul 17 05:29:21.848808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:21.848436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" event={"ID":"b0d425ad-1abd-4244-83cd-8fee4eaf68e6","Type":"ContainerStarted","Data":"4f57d4fbe7413a84d8a2db853eb2bc10e403e147e759115aa9c834a415eb2bff"}
Jul 17 05:29:21.848808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:21.848579    2582 scope.go:117] "RemoveContainer" containerID="3773967144b34b6a776280086a6b26f3c8e6f500b382fff384cc728acbeaf3f4"
Jul 17 05:29:22.854418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:22.854333    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerID="f910e7ee5f9677ae1dc94a10df0c158ff114e60b17c920545b5e744ca6ede7e7" exitCode=6
Jul 17 05:29:22.854418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:22.854409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" event={"ID":"b0d425ad-1abd-4244-83cd-8fee4eaf68e6","Type":"ContainerDied","Data":"f910e7ee5f9677ae1dc94a10df0c158ff114e60b17c920545b5e744ca6ede7e7"}
Jul 17 05:29:22.854867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:22.854462    2582 scope.go:117] "RemoveContainer" containerID="3773967144b34b6a776280086a6b26f3c8e6f500b382fff384cc728acbeaf3f4"
Jul 17 05:29:22.854867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:22.854636    2582 scope.go:117] "RemoveContainer" containerID="f910e7ee5f9677ae1dc94a10df0c158ff114e60b17c920545b5e744ca6ede7e7"
Jul 17 05:29:22.854867 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:22.854848    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6c78q_odh-ai-gateway-infra(b0d425ad-1abd-4244-83cd-8fee4eaf68e6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6"
Jul 17 05:29:23.859965 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:23.859942    2582 scope.go:117] "RemoveContainer" containerID="f910e7ee5f9677ae1dc94a10df0c158ff114e60b17c920545b5e744ca6ede7e7"
Jul 17 05:29:23.860359 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:23.860124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6c78q_odh-ai-gateway-infra(b0d425ad-1abd-4244-83cd-8fee4eaf68e6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6"
Jul 17 05:29:24.219045 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.219014    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"]
Jul 17 05:29:24.223513 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.223498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:24.229902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.229835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"]
Jul 17 05:29:24.254297 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.254242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a26ad31-1dcc-40c5-ad55-d64767e32d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2\" (UID: \"7a26ad31-1dcc-40c5-ad55-d64767e32d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:24.355193 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.355155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a26ad31-1dcc-40c5-ad55-d64767e32d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2\" (UID: \"7a26ad31-1dcc-40c5-ad55-d64767e32d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:24.355787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.355765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a26ad31-1dcc-40c5-ad55-d64767e32d80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2\" (UID: \"7a26ad31-1dcc-40c5-ad55-d64767e32d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:24.535991 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.535897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:24.664460 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.664443    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"]
Jul 17 05:29:24.666975 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:29:24.666949    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a26ad31_1dcc_40c5_ad55_d64767e32d80.slice/crio-5387e9c5b9cc658abc190140dc5baf01b48200bac123b3ae7160553c8847dfb4 WatchSource:0}: Error finding container 5387e9c5b9cc658abc190140dc5baf01b48200bac123b3ae7160553c8847dfb4: Status 404 returned error can't find the container with id 5387e9c5b9cc658abc190140dc5baf01b48200bac123b3ae7160553c8847dfb4
Jul 17 05:29:24.865625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:24.865535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" event={"ID":"7a26ad31-1dcc-40c5-ad55-d64767e32d80","Type":"ContainerStarted","Data":"5387e9c5b9cc658abc190140dc5baf01b48200bac123b3ae7160553c8847dfb4"}
Jul 17 05:29:25.871695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:25.871659    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerID="fdb54e2e2c4ff8c3aba505c93f797a19bbadea848c38df35a272cdd6009ed163" exitCode=7
Jul 17 05:29:25.872087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:25.871720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" event={"ID":"7a26ad31-1dcc-40c5-ad55-d64767e32d80","Type":"ContainerDied","Data":"fdb54e2e2c4ff8c3aba505c93f797a19bbadea848c38df35a272cdd6009ed163"}
Jul 17 05:29:25.872087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:25.871924    2582 scope.go:117] "RemoveContainer" containerID="fdb54e2e2c4ff8c3aba505c93f797a19bbadea848c38df35a272cdd6009ed163"
Jul 17 05:29:26.877501 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:26.877413    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerID="c8db94d0e9b7f53ddf5c5d92027a90633f8887bc53f28a69f62067ff33ededa8" exitCode=7
Jul 17 05:29:26.877501 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:26.877454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" event={"ID":"7a26ad31-1dcc-40c5-ad55-d64767e32d80","Type":"ContainerDied","Data":"c8db94d0e9b7f53ddf5c5d92027a90633f8887bc53f28a69f62067ff33ededa8"}
Jul 17 05:29:26.877971 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:26.877513    2582 scope.go:117] "RemoveContainer" containerID="fdb54e2e2c4ff8c3aba505c93f797a19bbadea848c38df35a272cdd6009ed163"
Jul 17 05:29:26.877971 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:26.877668    2582 scope.go:117] "RemoveContainer" containerID="c8db94d0e9b7f53ddf5c5d92027a90633f8887bc53f28a69f62067ff33ededa8"
Jul 17 05:29:26.877971 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:26.877882    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2_odh-ai-gateway-infra(7a26ad31-1dcc-40c5-ad55-d64767e32d80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80"
Jul 17 05:29:27.725464 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:27.725435    2582 scope.go:117] "RemoveContainer" containerID="f09666a9fb1f1ec6887d35df85628aea72ec7bccc74e26ed4c8c4a3e000721f4"
Jul 17 05:29:27.883937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:27.883909    2582 scope.go:117] "RemoveContainer" containerID="c8db94d0e9b7f53ddf5c5d92027a90633f8887bc53f28a69f62067ff33ededa8"
Jul 17 05:29:27.884372 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:27.884086    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2_odh-ai-gateway-infra(7a26ad31-1dcc-40c5-ad55-d64767e32d80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80"
Jul 17 05:29:28.889518 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:28.889485    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerID="0fdd0491fb342a06555fef3ae9f8ad229f8cb3ef575ab0c4d0a0a0d8cacb0967" exitCode=6
Jul 17 05:29:28.889962 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:28.889554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" event={"ID":"bdc8772c-ecbd-46ab-bc7e-d53c70e14867","Type":"ContainerDied","Data":"0fdd0491fb342a06555fef3ae9f8ad229f8cb3ef575ab0c4d0a0a0d8cacb0967"}
Jul 17 05:29:28.889962 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:28.889613    2582 scope.go:117] "RemoveContainer" containerID="f09666a9fb1f1ec6887d35df85628aea72ec7bccc74e26ed4c8c4a3e000721f4"
Jul 17 05:29:28.889962 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:28.889886    2582 scope.go:117] "RemoveContainer" containerID="0fdd0491fb342a06555fef3ae9f8ad229f8cb3ef575ab0c4d0a0a0d8cacb0967"
Jul 17 05:29:28.890096 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:28.890075    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp_odh-ai-gateway-infra(bdc8772c-ecbd-46ab-bc7e-d53c70e14867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867"
Jul 17 05:29:29.913532 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:29.913501    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"]
Jul 17 05:29:30.050875 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.050842    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:30.102989 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.102953    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc8772c-ecbd-46ab-bc7e-d53c70e14867-maas-api-service-ca\") pod \"bdc8772c-ecbd-46ab-bc7e-d53c70e14867\" (UID: \"bdc8772c-ecbd-46ab-bc7e-d53c70e14867\") "
Jul 17 05:29:30.103340 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.103320    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc8772c-ecbd-46ab-bc7e-d53c70e14867-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdc8772c-ecbd-46ab-bc7e-d53c70e14867" (UID: "bdc8772c-ecbd-46ab-bc7e-d53c70e14867"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:29:30.203772 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.203742    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc8772c-ecbd-46ab-bc7e-d53c70e14867-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:29:30.901152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.901120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp" event={"ID":"bdc8772c-ecbd-46ab-bc7e-d53c70e14867","Type":"ContainerDied","Data":"b756a39cea1189e8eb05316fb29dc946aaf2823ad062a94556af23e5e043305b"}
Jul 17 05:29:30.901152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.901139    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"
Jul 17 05:29:30.901453 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.901177    2582 scope.go:117] "RemoveContainer" containerID="0fdd0491fb342a06555fef3ae9f8ad229f8cb3ef575ab0c4d0a0a0d8cacb0967"
Jul 17 05:29:30.917213 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.917187    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"]
Jul 17 05:29:30.925489 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:30.925466    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-cwkcp"]
Jul 17 05:29:32.729339 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:32.729303    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" path="/var/lib/kubelet/pods/bdc8772c-ecbd-46ab-bc7e-d53c70e14867/volumes"
Jul 17 05:29:34.724911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:34.724879    2582 scope.go:117] "RemoveContainer" containerID="f910e7ee5f9677ae1dc94a10df0c158ff114e60b17c920545b5e744ca6ede7e7"
Jul 17 05:29:35.923702 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:35.923671    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerID="7a360351f13b7d9e2c1c22dc2f87abf11564414d515728eb49d7b583e1ece1ee" exitCode=6
Jul 17 05:29:35.924174 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:35.923710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" event={"ID":"b0d425ad-1abd-4244-83cd-8fee4eaf68e6","Type":"ContainerDied","Data":"7a360351f13b7d9e2c1c22dc2f87abf11564414d515728eb49d7b583e1ece1ee"}
Jul 17 05:29:35.924174 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:35.923757    2582 scope.go:117] "RemoveContainer" containerID="f910e7ee5f9677ae1dc94a10df0c158ff114e60b17c920545b5e744ca6ede7e7"
Jul 17 05:29:35.924174 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:35.924073    2582 scope.go:117] "RemoveContainer" containerID="7a360351f13b7d9e2c1c22dc2f87abf11564414d515728eb49d7b583e1ece1ee"
Jul 17 05:29:35.924353 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:35.924288    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6c78q_odh-ai-gateway-infra(b0d425ad-1abd-4244-83cd-8fee4eaf68e6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6"
Jul 17 05:29:36.951154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:36.951119    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"]
Jul 17 05:29:37.080841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.080817    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:37.163041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.163001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d425ad-1abd-4244-83cd-8fee4eaf68e6-maas-api-service-ca\") pod \"b0d425ad-1abd-4244-83cd-8fee4eaf68e6\" (UID: \"b0d425ad-1abd-4244-83cd-8fee4eaf68e6\") "
Jul 17 05:29:37.163419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.163394    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d425ad-1abd-4244-83cd-8fee4eaf68e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0d425ad-1abd-4244-83cd-8fee4eaf68e6" (UID: "b0d425ad-1abd-4244-83cd-8fee4eaf68e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:29:37.263644 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.263549    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d425ad-1abd-4244-83cd-8fee4eaf68e6-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:29:37.934725 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.934691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q" event={"ID":"b0d425ad-1abd-4244-83cd-8fee4eaf68e6","Type":"ContainerDied","Data":"4f57d4fbe7413a84d8a2db853eb2bc10e403e147e759115aa9c834a415eb2bff"}
Jul 17 05:29:37.934904 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.934729    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"
Jul 17 05:29:37.934904 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.934750    2582 scope.go:117] "RemoveContainer" containerID="7a360351f13b7d9e2c1c22dc2f87abf11564414d515728eb49d7b583e1ece1ee"
Jul 17 05:29:37.955364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.955336    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"]
Jul 17 05:29:37.961778 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:37.961755    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6c78q"]
Jul 17 05:29:38.729189 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:38.729152    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" path="/var/lib/kubelet/pods/b0d425ad-1abd-4244-83cd-8fee4eaf68e6/volumes"
Jul 17 05:29:40.724885 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:40.724854    2582 scope.go:117] "RemoveContainer" containerID="c8db94d0e9b7f53ddf5c5d92027a90633f8887bc53f28a69f62067ff33ededa8"
Jul 17 05:29:41.954114 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:41.954087    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerID="20d868ebcb977f70308668d57949e6bad2742a627c34fc9f81205305b6c61eae" exitCode=7
Jul 17 05:29:41.954568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:41.954163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" event={"ID":"7a26ad31-1dcc-40c5-ad55-d64767e32d80","Type":"ContainerDied","Data":"20d868ebcb977f70308668d57949e6bad2742a627c34fc9f81205305b6c61eae"}
Jul 17 05:29:41.954568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:41.954224    2582 scope.go:117] "RemoveContainer" containerID="c8db94d0e9b7f53ddf5c5d92027a90633f8887bc53f28a69f62067ff33ededa8"
Jul 17 05:29:41.954568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:41.954520    2582 scope.go:117] "RemoveContainer" containerID="20d868ebcb977f70308668d57949e6bad2742a627c34fc9f81205305b6c61eae"
Jul 17 05:29:41.954752 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:41.954703    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2_odh-ai-gateway-infra(7a26ad31-1dcc-40c5-ad55-d64767e32d80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80"
Jul 17 05:29:42.979227 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:42.979191    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"]
Jul 17 05:29:43.114112 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.114087    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:43.217371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.217325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a26ad31-1dcc-40c5-ad55-d64767e32d80-maas-api-service-ca\") pod \"7a26ad31-1dcc-40c5-ad55-d64767e32d80\" (UID: \"7a26ad31-1dcc-40c5-ad55-d64767e32d80\") "
Jul 17 05:29:43.217742 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.217713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a26ad31-1dcc-40c5-ad55-d64767e32d80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a26ad31-1dcc-40c5-ad55-d64767e32d80" (UID: "7a26ad31-1dcc-40c5-ad55-d64767e32d80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:29:43.318417 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.318329    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a26ad31-1dcc-40c5-ad55-d64767e32d80-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:29:43.965322 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.965287    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2" event={"ID":"7a26ad31-1dcc-40c5-ad55-d64767e32d80","Type":"ContainerDied","Data":"5387e9c5b9cc658abc190140dc5baf01b48200bac123b3ae7160553c8847dfb4"}
Jul 17 05:29:43.965322 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.965303    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"
Jul 17 05:29:43.965559 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.965348    2582 scope.go:117] "RemoveContainer" containerID="20d868ebcb977f70308668d57949e6bad2742a627c34fc9f81205305b6c61eae"
Jul 17 05:29:43.985495 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.985469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"]
Jul 17 05:29:43.991439 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:43.991416    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ztgv2"]
Jul 17 05:29:44.729877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:44.729839    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" path="/var/lib/kubelet/pods/7a26ad31-1dcc-40c5-ad55-d64767e32d80/volumes"
Jul 17 05:29:46.908718 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.908676    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"]
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909100    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909119    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909130    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909138    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909150    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909157    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909177    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909185    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909193    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909200    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909208    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909216    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909225    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909234    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909244    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909252    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909360    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909371    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909381    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909389    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909398    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:29:46.909816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.909407    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:29:46.913808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.913785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:29:46.916402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.916380    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:29:46.916538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.916412    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:29:46.920812 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:46.920789    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"]
Jul 17 05:29:47.050912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.050870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65885c3b-5150-4e0e-a713-9cda5501f586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt\" (UID: \"65885c3b-5150-4e0e-a713-9cda5501f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:29:47.151755 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.151713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65885c3b-5150-4e0e-a713-9cda5501f586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt\" (UID: \"65885c3b-5150-4e0e-a713-9cda5501f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:29:47.152386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.152366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65885c3b-5150-4e0e-a713-9cda5501f586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt\" (UID: \"65885c3b-5150-4e0e-a713-9cda5501f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:29:47.227096 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.227067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:29:47.361341 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.361307    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"]
Jul 17 05:29:47.364663 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:29:47.364640    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65885c3b_5150_4e0e_a713_9cda5501f586.slice/crio-5772760f05957ed24120908949c4c37c2d2ce132e7ff9d323f20b7649576442f WatchSource:0}: Error finding container 5772760f05957ed24120908949c4c37c2d2ce132e7ff9d323f20b7649576442f: Status 404 returned error can't find the container with id 5772760f05957ed24120908949c4c37c2d2ce132e7ff9d323f20b7649576442f
Jul 17 05:29:47.984188 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.984152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerStarted","Data":"64f8acb701ceb5339bb8d7ea28c77b37192d1b60265d78d70d50cbbe95a611d6"}
Jul 17 05:29:47.984640 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:47.984219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerStarted","Data":"5772760f05957ed24120908949c4c37c2d2ce132e7ff9d323f20b7649576442f"}
Jul 17 05:29:48.000742 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:48.000699    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" podStartSLOduration=2.000672546 podStartE2EDuration="2.000672546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:29:47.999531825 +0000 UTC m=+2391.736594187" watchObservedRunningTime="2026-07-17 05:29:48.000672546 +0000 UTC m=+2391.737734882"
Jul 17 05:29:48.989107 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:48.989070    2582 generic.go:356] "Generic (PLEG): container finished" podID="65885c3b-5150-4e0e-a713-9cda5501f586" containerID="64f8acb701ceb5339bb8d7ea28c77b37192d1b60265d78d70d50cbbe95a611d6" exitCode=6
Jul 17 05:29:48.989515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:48.989168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerDied","Data":"64f8acb701ceb5339bb8d7ea28c77b37192d1b60265d78d70d50cbbe95a611d6"}
Jul 17 05:29:48.989515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:48.989332    2582 scope.go:117] "RemoveContainer" containerID="64f8acb701ceb5339bb8d7ea28c77b37192d1b60265d78d70d50cbbe95a611d6"
Jul 17 05:29:49.995749 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:49.995719    2582 generic.go:356] "Generic (PLEG): container finished" podID="65885c3b-5150-4e0e-a713-9cda5501f586" containerID="b40358252d8841880a69133a741783da11df41e469bf4135a62c45183d4c9adb" exitCode=6
Jul 17 05:29:49.996147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:49.995812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerDied","Data":"b40358252d8841880a69133a741783da11df41e469bf4135a62c45183d4c9adb"}
Jul 17 05:29:49.996147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:49.995905    2582 scope.go:117] "RemoveContainer" containerID="64f8acb701ceb5339bb8d7ea28c77b37192d1b60265d78d70d50cbbe95a611d6"
Jul 17 05:29:49.996147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:49.996140    2582 scope.go:117] "RemoveContainer" containerID="b40358252d8841880a69133a741783da11df41e469bf4135a62c45183d4c9adb"
Jul 17 05:29:49.996379 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:49.996360    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt_odh-ai-gateway-infra(65885c3b-5150-4e0e-a713-9cda5501f586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" podUID="65885c3b-5150-4e0e-a713-9cda5501f586"
Jul 17 05:29:51.001155 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:51.001127    2582 scope.go:117] "RemoveContainer" containerID="b40358252d8841880a69133a741783da11df41e469bf4135a62c45183d4c9adb"
Jul 17 05:29:51.001607 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:29:51.001344    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt_odh-ai-gateway-infra(65885c3b-5150-4e0e-a713-9cda5501f586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" podUID="65885c3b-5150-4e0e-a713-9cda5501f586"
Jul 17 05:29:56.807820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:56.807788    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:29:56.812137 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:29:56.812114    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:30:00.139386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.139347    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"]
Jul 17 05:30:00.139861 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.139830    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:30:00.139861 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.139850    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:30:00.139982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.139919    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d425ad-1abd-4244-83cd-8fee4eaf68e6" containerName="revoke-keys"
Jul 17 05:30:00.139982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.139935    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a26ad31-1dcc-40c5-ad55-d64767e32d80" containerName="revoke-keys"
Jul 17 05:30:00.139982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.139944    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc8772c-ecbd-46ab-bc7e-d53c70e14867" containerName="revoke-keys"
Jul 17 05:30:00.144845 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.144821    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"]
Jul 17 05:30:00.144999 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.144935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"
Jul 17 05:30:00.147949 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.147926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"
Jul 17 05:30:00.149367 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.149342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"]
Jul 17 05:30:00.152390 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.152367    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"]
Jul 17 05:30:00.157953 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.157925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"
Jul 17 05:30:00.165170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.165147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"
Jul 17 05:30:00.311485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.311462    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"]
Jul 17 05:30:00.313893 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:30:00.313856    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0662ef5_0f74_4fb5_b7c4_ce20800c2031.slice/crio-74fa960e4f0de186bb883299b3e2431eec333ec11f02531cb4fce507e6894dcc WatchSource:0}: Error finding container 74fa960e4f0de186bb883299b3e2431eec333ec11f02531cb4fce507e6894dcc: Status 404 returned error can't find the container with id 74fa960e4f0de186bb883299b3e2431eec333ec11f02531cb4fce507e6894dcc
Jul 17 05:30:00.327963 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:00.327938    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"]
Jul 17 05:30:00.330359 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:30:00.330331    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33058111_4c95_481d_bf2c_69dc079f2644.slice/crio-63bb06fdfab1137044aa7f118296e02f520cb6558b468205bab07e94f1faef60 WatchSource:0}: Error finding container 63bb06fdfab1137044aa7f118296e02f520cb6558b468205bab07e94f1faef60: Status 404 returned error can't find the container with id 63bb06fdfab1137044aa7f118296e02f520cb6558b468205bab07e94f1faef60
Jul 17 05:30:01.043076 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.043038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d" event={"ID":"c0662ef5-0f74-4fb5-b7c4-ce20800c2031","Type":"ContainerStarted","Data":"8e583a6b62ae6d93086c2f52ee503977f40182d36aa23baaac7bcd98895d448f"}
Jul 17 05:30:01.043299 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.043102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d" event={"ID":"c0662ef5-0f74-4fb5-b7c4-ce20800c2031","Type":"ContainerStarted","Data":"74fa960e4f0de186bb883299b3e2431eec333ec11f02531cb4fce507e6894dcc"}
Jul 17 05:30:01.044543 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.044516    2582 generic.go:356] "Generic (PLEG): container finished" podID="33058111-4c95-481d-bf2c-69dc079f2644" containerID="1354a1c34d2219cc46bac9fec0cb6359f832dc5abf119a2c221933d5375383fe" exitCode=7
Jul 17 05:30:01.044655 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.044564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" event={"ID":"33058111-4c95-481d-bf2c-69dc079f2644","Type":"ContainerDied","Data":"1354a1c34d2219cc46bac9fec0cb6359f832dc5abf119a2c221933d5375383fe"}
Jul 17 05:30:01.044655 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.044609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" event={"ID":"33058111-4c95-481d-bf2c-69dc079f2644","Type":"ContainerStarted","Data":"63bb06fdfab1137044aa7f118296e02f520cb6558b468205bab07e94f1faef60"}
Jul 17 05:30:01.044757 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.044701    2582 scope.go:117] "RemoveContainer" containerID="1354a1c34d2219cc46bac9fec0cb6359f832dc5abf119a2c221933d5375383fe"
Jul 17 05:30:01.060919 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.060872    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d" podStartSLOduration=1.060857232 podStartE2EDuration="1.060857232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:30:01.059669263 +0000 UTC m=+2404.796731601" watchObservedRunningTime="2026-07-17 05:30:01.060857232 +0000 UTC m=+2404.797919622"
Jul 17 05:30:01.724777 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:01.724747    2582 scope.go:117] "RemoveContainer" containerID="b40358252d8841880a69133a741783da11df41e469bf4135a62c45183d4c9adb"
Jul 17 05:30:02.050252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.050218    2582 generic.go:356] "Generic (PLEG): container finished" podID="c0662ef5-0f74-4fb5-b7c4-ce20800c2031" containerID="8e583a6b62ae6d93086c2f52ee503977f40182d36aa23baaac7bcd98895d448f" exitCode=0
Jul 17 05:30:02.050465 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.050314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d" event={"ID":"c0662ef5-0f74-4fb5-b7c4-ce20800c2031","Type":"ContainerDied","Data":"8e583a6b62ae6d93086c2f52ee503977f40182d36aa23baaac7bcd98895d448f"}
Jul 17 05:30:02.052057 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.052030    2582 generic.go:356] "Generic (PLEG): container finished" podID="33058111-4c95-481d-bf2c-69dc079f2644" containerID="cefc1c3e881d4ff66c386aa49dc3f46aa5b0070b576d2ab44c3b30e89080cab7" exitCode=7
Jul 17 05:30:02.052192 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.052117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" event={"ID":"33058111-4c95-481d-bf2c-69dc079f2644","Type":"ContainerDied","Data":"cefc1c3e881d4ff66c386aa49dc3f46aa5b0070b576d2ab44c3b30e89080cab7"}
Jul 17 05:30:02.052192 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.052184    2582 scope.go:117] "RemoveContainer" containerID="1354a1c34d2219cc46bac9fec0cb6359f832dc5abf119a2c221933d5375383fe"
Jul 17 05:30:02.052325 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.052297    2582 scope.go:117] "RemoveContainer" containerID="cefc1c3e881d4ff66c386aa49dc3f46aa5b0070b576d2ab44c3b30e89080cab7"
Jul 17 05:30:02.052557 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:02.052536    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8_odh-ai-gateway-infra(33058111-4c95-481d-bf2c-69dc079f2644)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" podUID="33058111-4c95-481d-bf2c-69dc079f2644"
Jul 17 05:30:02.053914 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:02.053896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerStarted","Data":"99140e2c6e5fbb4c717456325901ea9e445e00be3dd7c5d193697c2e74d667e6"}
Jul 17 05:30:03.065311 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.065253    2582 scope.go:117] "RemoveContainer" containerID="cefc1c3e881d4ff66c386aa49dc3f46aa5b0070b576d2ab44c3b30e89080cab7"
Jul 17 05:30:03.065764 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:03.065488    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8_odh-ai-gateway-infra(33058111-4c95-481d-bf2c-69dc079f2644)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" podUID="33058111-4c95-481d-bf2c-69dc079f2644"
Jul 17 05:30:03.066862 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.066841    2582 generic.go:356] "Generic (PLEG): container finished" podID="65885c3b-5150-4e0e-a713-9cda5501f586" containerID="99140e2c6e5fbb4c717456325901ea9e445e00be3dd7c5d193697c2e74d667e6" exitCode=6
Jul 17 05:30:03.066948 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.066910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerDied","Data":"99140e2c6e5fbb4c717456325901ea9e445e00be3dd7c5d193697c2e74d667e6"}
Jul 17 05:30:03.067004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.066992    2582 scope.go:117] "RemoveContainer" containerID="b40358252d8841880a69133a741783da11df41e469bf4135a62c45183d4c9adb"
Jul 17 05:30:03.067529 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.067251    2582 scope.go:117] "RemoveContainer" containerID="99140e2c6e5fbb4c717456325901ea9e445e00be3dd7c5d193697c2e74d667e6"
Jul 17 05:30:03.067529 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:03.067470    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt_odh-ai-gateway-infra(65885c3b-5150-4e0e-a713-9cda5501f586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" podUID="65885c3b-5150-4e0e-a713-9cda5501f586"
Jul 17 05:30:03.094999 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.094960    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"]
Jul 17 05:30:03.197801 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:03.197778    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"
Jul 17 05:30:04.073721 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.073684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d" event={"ID":"c0662ef5-0f74-4fb5-b7c4-ce20800c2031","Type":"ContainerDied","Data":"74fa960e4f0de186bb883299b3e2431eec333ec11f02531cb4fce507e6894dcc"}
Jul 17 05:30:04.074128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.073734    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74fa960e4f0de186bb883299b3e2431eec333ec11f02531cb4fce507e6894dcc"
Jul 17 05:30:04.074128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.073699    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737770-7652d"
Jul 17 05:30:04.203424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.203398    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:30:04.297867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.297832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65885c3b-5150-4e0e-a713-9cda5501f586-maas-api-service-ca\") pod \"65885c3b-5150-4e0e-a713-9cda5501f586\" (UID: \"65885c3b-5150-4e0e-a713-9cda5501f586\") "
Jul 17 05:30:04.298227 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.298204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65885c3b-5150-4e0e-a713-9cda5501f586-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65885c3b-5150-4e0e-a713-9cda5501f586" (UID: "65885c3b-5150-4e0e-a713-9cda5501f586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:30:04.398804 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:04.398714    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65885c3b-5150-4e0e-a713-9cda5501f586-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:30:05.079595 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:05.079518    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"
Jul 17 05:30:05.079998 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:05.079514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt" event={"ID":"65885c3b-5150-4e0e-a713-9cda5501f586","Type":"ContainerDied","Data":"5772760f05957ed24120908949c4c37c2d2ce132e7ff9d323f20b7649576442f"}
Jul 17 05:30:05.079998 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:05.079649    2582 scope.go:117] "RemoveContainer" containerID="99140e2c6e5fbb4c717456325901ea9e445e00be3dd7c5d193697c2e74d667e6"
Jul 17 05:30:05.095856 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:05.095830    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"]
Jul 17 05:30:05.100027 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:05.100002    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bjfzt"]
Jul 17 05:30:06.731718 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:06.731685    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" path="/var/lib/kubelet/pods/65885c3b-5150-4e0e-a713-9cda5501f586/volumes"
Jul 17 05:30:08.854422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854378    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"]
Jul 17 05:30:08.854924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854785    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0662ef5-0f74-4fb5-b7c4-ce20800c2031" containerName="cleanup"
Jul 17 05:30:08.854924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854802    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0662ef5-0f74-4fb5-b7c4-ce20800c2031" containerName="cleanup"
Jul 17 05:30:08.854924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854819    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:08.854924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854827    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:08.854924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854853    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:08.854924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854862    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:08.855233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854930    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0662ef5-0f74-4fb5-b7c4-ce20800c2031" containerName="cleanup"
Jul 17 05:30:08.855233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854941    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:08.855233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.854953    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:08.859341 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.859319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:08.862267 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.862231    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:30:08.865056 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:08.865032    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"]
Jul 17 05:30:09.037703 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:09.037668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d954536c-3bc5-46c9-9633-2a0ca7ee9366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9\" (UID: \"d954536c-3bc5-46c9-9633-2a0ca7ee9366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:09.138467 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:09.138379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d954536c-3bc5-46c9-9633-2a0ca7ee9366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9\" (UID: \"d954536c-3bc5-46c9-9633-2a0ca7ee9366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:09.138976 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:09.138956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d954536c-3bc5-46c9-9633-2a0ca7ee9366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9\" (UID: \"d954536c-3bc5-46c9-9633-2a0ca7ee9366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:09.170406 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:09.170378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:09.302839 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:09.302796    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"]
Jul 17 05:30:09.305810 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:30:09.305782    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd954536c_3bc5_46c9_9633_2a0ca7ee9366.slice/crio-971b7663f22744d6dcaf04223ddd60fc51e8fb2aa49f2a0bd547e5ac267d4004 WatchSource:0}: Error finding container 971b7663f22744d6dcaf04223ddd60fc51e8fb2aa49f2a0bd547e5ac267d4004: Status 404 returned error can't find the container with id 971b7663f22744d6dcaf04223ddd60fc51e8fb2aa49f2a0bd547e5ac267d4004
Jul 17 05:30:10.103955 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.103920    2582 generic.go:356] "Generic (PLEG): container finished" podID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerID="cb516c3f553844ebf94c3f96702d54a4e4773ff4dff931e12b4594c58fa596d6" exitCode=6
Jul 17 05:30:10.104360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.103988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" event={"ID":"d954536c-3bc5-46c9-9633-2a0ca7ee9366","Type":"ContainerDied","Data":"cb516c3f553844ebf94c3f96702d54a4e4773ff4dff931e12b4594c58fa596d6"}
Jul 17 05:30:10.104360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.104034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" event={"ID":"d954536c-3bc5-46c9-9633-2a0ca7ee9366","Type":"ContainerStarted","Data":"971b7663f22744d6dcaf04223ddd60fc51e8fb2aa49f2a0bd547e5ac267d4004"}
Jul 17 05:30:10.104360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.104229    2582 scope.go:117] "RemoveContainer" containerID="cb516c3f553844ebf94c3f96702d54a4e4773ff4dff931e12b4594c58fa596d6"
Jul 17 05:30:10.736585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.736557    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"]
Jul 17 05:30:10.736932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.736916    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:10.737011 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.736935    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:10.737070 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.737022    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65885c3b-5150-4e0e-a713-9cda5501f586" containerName="revoke-keys"
Jul 17 05:30:10.739853 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.739833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:10.747208 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.747185    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"]
Jul 17 05:30:10.851418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.851377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9aa6-8414-4848-9984-4a3f3742542c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-rs47p\" (UID: \"1b3c9aa6-8414-4848-9984-4a3f3742542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:10.952105 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.952065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9aa6-8414-4848-9984-4a3f3742542c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-rs47p\" (UID: \"1b3c9aa6-8414-4848-9984-4a3f3742542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:10.952731 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:10.952711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9aa6-8414-4848-9984-4a3f3742542c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-rs47p\" (UID: \"1b3c9aa6-8414-4848-9984-4a3f3742542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:11.068777 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:11.068742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:11.111768 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:11.111735    2582 generic.go:356] "Generic (PLEG): container finished" podID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerID="e3ed5fad530f179478e26a59e585913e26a491fb045a4858deee9e97f71585e6" exitCode=6
Jul 17 05:30:11.112186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:11.111790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" event={"ID":"d954536c-3bc5-46c9-9633-2a0ca7ee9366","Type":"ContainerDied","Data":"e3ed5fad530f179478e26a59e585913e26a491fb045a4858deee9e97f71585e6"}
Jul 17 05:30:11.112186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:11.111865    2582 scope.go:117] "RemoveContainer" containerID="cb516c3f553844ebf94c3f96702d54a4e4773ff4dff931e12b4594c58fa596d6"
Jul 17 05:30:11.112186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:11.112081    2582 scope.go:117] "RemoveContainer" containerID="e3ed5fad530f179478e26a59e585913e26a491fb045a4858deee9e97f71585e6"
Jul 17 05:30:11.112401 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:11.112345    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9_odh-ai-gateway-infra(d954536c-3bc5-46c9-9633-2a0ca7ee9366)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366"
Jul 17 05:30:11.204951 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:11.204923    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"]
Jul 17 05:30:11.207455 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:30:11.207429    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3c9aa6_8414_4848_9984_4a3f3742542c.slice/crio-1519184a1a16a832add5482a77264d7f30e56a2dded12565fea7d69adca5a42d WatchSource:0}: Error finding container 1519184a1a16a832add5482a77264d7f30e56a2dded12565fea7d69adca5a42d: Status 404 returned error can't find the container with id 1519184a1a16a832add5482a77264d7f30e56a2dded12565fea7d69adca5a42d
Jul 17 05:30:12.117875 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:12.117791    2582 scope.go:117] "RemoveContainer" containerID="e3ed5fad530f179478e26a59e585913e26a491fb045a4858deee9e97f71585e6"
Jul 17 05:30:12.118333 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:12.118017    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9_odh-ai-gateway-infra(d954536c-3bc5-46c9-9633-2a0ca7ee9366)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366"
Jul 17 05:30:12.119114 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:12.119094    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerID="131fcef1ff39a330991f45593a9fafdb429884ea8322573864ca23fcf9eda680" exitCode=6
Jul 17 05:30:12.119203 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:12.119120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" event={"ID":"1b3c9aa6-8414-4848-9984-4a3f3742542c","Type":"ContainerDied","Data":"131fcef1ff39a330991f45593a9fafdb429884ea8322573864ca23fcf9eda680"}
Jul 17 05:30:12.119203 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:12.119163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" event={"ID":"1b3c9aa6-8414-4848-9984-4a3f3742542c","Type":"ContainerStarted","Data":"1519184a1a16a832add5482a77264d7f30e56a2dded12565fea7d69adca5a42d"}
Jul 17 05:30:12.119316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:12.119308    2582 scope.go:117] "RemoveContainer" containerID="131fcef1ff39a330991f45593a9fafdb429884ea8322573864ca23fcf9eda680"
Jul 17 05:30:13.125512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:13.125422    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerID="2d681e02bf4e218d5971c3c3a5c33c81f8d0b66201fe73d9286fcb1adb33b4ae" exitCode=6
Jul 17 05:30:13.125907 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:13.125503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" event={"ID":"1b3c9aa6-8414-4848-9984-4a3f3742542c","Type":"ContainerDied","Data":"2d681e02bf4e218d5971c3c3a5c33c81f8d0b66201fe73d9286fcb1adb33b4ae"}
Jul 17 05:30:13.125907 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:13.125558    2582 scope.go:117] "RemoveContainer" containerID="131fcef1ff39a330991f45593a9fafdb429884ea8322573864ca23fcf9eda680"
Jul 17 05:30:13.125907 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:13.125656    2582 scope.go:117] "RemoveContainer" containerID="2d681e02bf4e218d5971c3c3a5c33c81f8d0b66201fe73d9286fcb1adb33b4ae"
Jul 17 05:30:13.125907 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:13.125882    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-rs47p_odh-ai-gateway-infra(1b3c9aa6-8414-4848-9984-4a3f3742542c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c"
Jul 17 05:30:14.131393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.131367    2582 scope.go:117] "RemoveContainer" containerID="2d681e02bf4e218d5971c3c3a5c33c81f8d0b66201fe73d9286fcb1adb33b4ae"
Jul 17 05:30:14.131771 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:14.131566    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-rs47p_odh-ai-gateway-infra(1b3c9aa6-8414-4848-9984-4a3f3742542c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c"
Jul 17 05:30:14.259063 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.259025    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"]
Jul 17 05:30:14.263352 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.263337    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:14.268884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.268847    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"]
Jul 17 05:30:14.382053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.381959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df95432-6e9a-410e-96f4-85fa0b09d4e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2\" (UID: \"7df95432-6e9a-410e-96f4-85fa0b09d4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:14.483104 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.483056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df95432-6e9a-410e-96f4-85fa0b09d4e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2\" (UID: \"7df95432-6e9a-410e-96f4-85fa0b09d4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:14.483650 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.483632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df95432-6e9a-410e-96f4-85fa0b09d4e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2\" (UID: \"7df95432-6e9a-410e-96f4-85fa0b09d4e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:14.575237 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.575203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:14.707376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:14.707343    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"]
Jul 17 05:30:14.709875 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:30:14.709849    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df95432_6e9a_410e_96f4_85fa0b09d4e7.slice/crio-2bcd777b7843f00325970ee713af9c8bd97c09a42c620ff390aed97968f8f9e9 WatchSource:0}: Error finding container 2bcd777b7843f00325970ee713af9c8bd97c09a42c620ff390aed97968f8f9e9: Status 404 returned error can't find the container with id 2bcd777b7843f00325970ee713af9c8bd97c09a42c620ff390aed97968f8f9e9
Jul 17 05:30:15.137540 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:15.137505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerStarted","Data":"4c55592bc6dae92308ed13f2619ba88eeb07e25f89b176aadc517ffcae2d8640"}
Jul 17 05:30:15.137883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:15.137571    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerStarted","Data":"2bcd777b7843f00325970ee713af9c8bd97c09a42c620ff390aed97968f8f9e9"}
Jul 17 05:30:15.152035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:15.151969    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" podStartSLOduration=1.151948112 podStartE2EDuration="1.151948112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:30:15.151658925 +0000 UTC m=+2418.888721260" watchObservedRunningTime="2026-07-17 05:30:15.151948112 +0000 UTC m=+2418.889010449"
Jul 17 05:30:16.144100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:16.144063    2582 generic.go:356] "Generic (PLEG): container finished" podID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerID="4c55592bc6dae92308ed13f2619ba88eeb07e25f89b176aadc517ffcae2d8640" exitCode=7
Jul 17 05:30:16.144871 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:16.144137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerDied","Data":"4c55592bc6dae92308ed13f2619ba88eeb07e25f89b176aadc517ffcae2d8640"}
Jul 17 05:30:16.144871 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:16.144375    2582 scope.go:117] "RemoveContainer" containerID="4c55592bc6dae92308ed13f2619ba88eeb07e25f89b176aadc517ffcae2d8640"
Jul 17 05:30:16.726876 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:16.726843    2582 scope.go:117] "RemoveContainer" containerID="cefc1c3e881d4ff66c386aa49dc3f46aa5b0070b576d2ab44c3b30e89080cab7"
Jul 17 05:30:17.150334 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:17.150304    2582 generic.go:356] "Generic (PLEG): container finished" podID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerID="369d2a82da600996c3c7d46851b0be907c6cd8d9c2a04c43f37d220bb49af061" exitCode=7
Jul 17 05:30:17.150807 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:17.150392    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerDied","Data":"369d2a82da600996c3c7d46851b0be907c6cd8d9c2a04c43f37d220bb49af061"}
Jul 17 05:30:17.150807 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:17.150451    2582 scope.go:117] "RemoveContainer" containerID="4c55592bc6dae92308ed13f2619ba88eeb07e25f89b176aadc517ffcae2d8640"
Jul 17 05:30:17.150807 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:17.150637    2582 scope.go:117] "RemoveContainer" containerID="369d2a82da600996c3c7d46851b0be907c6cd8d9c2a04c43f37d220bb49af061"
Jul 17 05:30:17.150946 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:17.150877    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2_odh-ai-gateway-infra(7df95432-6e9a-410e-96f4-85fa0b09d4e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7"
Jul 17 05:30:17.152310 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:17.152288    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" event={"ID":"33058111-4c95-481d-bf2c-69dc079f2644","Type":"ContainerStarted","Data":"4b7cbcd9dc397c0c00abab8d4e0a425f123d8a3479355c9270b5f590654ddaef"}
Jul 17 05:30:17.186436 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:17.180965    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" podStartSLOduration=17.180943938 podStartE2EDuration="17.180943938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:30:17.179891168 +0000 UTC m=+2420.916953530" watchObservedRunningTime="2026-07-17 05:30:17.180943938 +0000 UTC m=+2420.918006276"
Jul 17 05:30:18.159040 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:18.159010    2582 scope.go:117] "RemoveContainer" containerID="369d2a82da600996c3c7d46851b0be907c6cd8d9c2a04c43f37d220bb49af061"
Jul 17 05:30:18.159497 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:18.159231    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2_odh-ai-gateway-infra(7df95432-6e9a-410e-96f4-85fa0b09d4e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7"
Jul 17 05:30:18.160531 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:18.160510    2582 generic.go:356] "Generic (PLEG): container finished" podID="33058111-4c95-481d-bf2c-69dc079f2644" containerID="4b7cbcd9dc397c0c00abab8d4e0a425f123d8a3479355c9270b5f590654ddaef" exitCode=7
Jul 17 05:30:18.160580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:18.160548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" event={"ID":"33058111-4c95-481d-bf2c-69dc079f2644","Type":"ContainerDied","Data":"4b7cbcd9dc397c0c00abab8d4e0a425f123d8a3479355c9270b5f590654ddaef"}
Jul 17 05:30:18.160628 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:18.160618    2582 scope.go:117] "RemoveContainer" containerID="cefc1c3e881d4ff66c386aa49dc3f46aa5b0070b576d2ab44c3b30e89080cab7"
Jul 17 05:30:18.160892 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:18.160872    2582 scope.go:117] "RemoveContainer" containerID="4b7cbcd9dc397c0c00abab8d4e0a425f123d8a3479355c9270b5f590654ddaef"
Jul 17 05:30:18.161084 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:18.161062    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8_odh-ai-gateway-infra(33058111-4c95-481d-bf2c-69dc079f2644)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" podUID="33058111-4c95-481d-bf2c-69dc079f2644"
Jul 17 05:30:18.189404 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:18.189378    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"]
Jul 17 05:30:19.296468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:19.296442    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"
Jul 17 05:30:20.172536 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:20.172508    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"
Jul 17 05:30:20.172536 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:20.172531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8" event={"ID":"33058111-4c95-481d-bf2c-69dc079f2644","Type":"ContainerDied","Data":"63bb06fdfab1137044aa7f118296e02f520cb6558b468205bab07e94f1faef60"}
Jul 17 05:30:20.172782 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:20.172588    2582 scope.go:117] "RemoveContainer" containerID="4b7cbcd9dc397c0c00abab8d4e0a425f123d8a3479355c9270b5f590654ddaef"
Jul 17 05:30:20.193298 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:20.193249    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"]
Jul 17 05:30:20.195954 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:20.195931    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf0ad9db-29737770-5qhb8"]
Jul 17 05:30:20.729549 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:20.729517    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33058111-4c95-481d-bf2c-69dc079f2644" path="/var/lib/kubelet/pods/33058111-4c95-481d-bf2c-69dc079f2644/volumes"
Jul 17 05:30:25.724857 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:25.724829    2582 scope.go:117] "RemoveContainer" containerID="e3ed5fad530f179478e26a59e585913e26a491fb045a4858deee9e97f71585e6"
Jul 17 05:30:26.200808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:26.200775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" event={"ID":"d954536c-3bc5-46c9-9633-2a0ca7ee9366","Type":"ContainerStarted","Data":"111f97255d8269099f402c5438bee727dbf67747bd41afedf829b0d2cbe48c99"}
Jul 17 05:30:26.218577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:26.218531    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" podStartSLOduration=18.218516126 podStartE2EDuration="18.218516126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:30:26.216914015 +0000 UTC m=+2429.953976353" watchObservedRunningTime="2026-07-17 05:30:26.218516126 +0000 UTC m=+2429.955578462"
Jul 17 05:30:27.206401 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:27.206370    2582 generic.go:356] "Generic (PLEG): container finished" podID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerID="111f97255d8269099f402c5438bee727dbf67747bd41afedf829b0d2cbe48c99" exitCode=6
Jul 17 05:30:27.206829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:27.206448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" event={"ID":"d954536c-3bc5-46c9-9633-2a0ca7ee9366","Type":"ContainerDied","Data":"111f97255d8269099f402c5438bee727dbf67747bd41afedf829b0d2cbe48c99"}
Jul 17 05:30:27.206829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:27.206507    2582 scope.go:117] "RemoveContainer" containerID="e3ed5fad530f179478e26a59e585913e26a491fb045a4858deee9e97f71585e6"
Jul 17 05:30:27.206829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:27.206799    2582 scope.go:117] "RemoveContainer" containerID="111f97255d8269099f402c5438bee727dbf67747bd41afedf829b0d2cbe48c99"
Jul 17 05:30:27.207022 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:27.207002    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9_odh-ai-gateway-infra(d954536c-3bc5-46c9-9633-2a0ca7ee9366)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366"
Jul 17 05:30:27.232943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:27.232915    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"]
Jul 17 05:30:27.725315 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:27.725285    2582 scope.go:117] "RemoveContainer" containerID="2d681e02bf4e218d5971c3c3a5c33c81f8d0b66201fe73d9286fcb1adb33b4ae"
Jul 17 05:30:28.213873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.213842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" event={"ID":"1b3c9aa6-8414-4848-9984-4a3f3742542c","Type":"ContainerStarted","Data":"5df373756b3bc9d4f9f47a3ed2f163ed65be1e5360fd148b20b2c16595bbe8c1"}
Jul 17 05:30:28.247183 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.247121    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" podStartSLOduration=18.247101354 podStartE2EDuration="18.247101354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:30:28.245962173 +0000 UTC m=+2431.983024510" watchObservedRunningTime="2026-07-17 05:30:28.247101354 +0000 UTC m=+2431.984163691"
Jul 17 05:30:28.341027 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.341003    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:28.393620 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.393592    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d954536c-3bc5-46c9-9633-2a0ca7ee9366-maas-api-service-ca\") pod \"d954536c-3bc5-46c9-9633-2a0ca7ee9366\" (UID: \"d954536c-3bc5-46c9-9633-2a0ca7ee9366\") "
Jul 17 05:30:28.393912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.393886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d954536c-3bc5-46c9-9633-2a0ca7ee9366-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d954536c-3bc5-46c9-9633-2a0ca7ee9366" (UID: "d954536c-3bc5-46c9-9633-2a0ca7ee9366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:30:28.495004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.494915    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d954536c-3bc5-46c9-9633-2a0ca7ee9366-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:30:28.724666 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:28.724634    2582 scope.go:117] "RemoveContainer" containerID="369d2a82da600996c3c7d46851b0be907c6cd8d9c2a04c43f37d220bb49af061"
Jul 17 05:30:29.219953 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.219922    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerID="5df373756b3bc9d4f9f47a3ed2f163ed65be1e5360fd148b20b2c16595bbe8c1" exitCode=6
Jul 17 05:30:29.220447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.220001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" event={"ID":"1b3c9aa6-8414-4848-9984-4a3f3742542c","Type":"ContainerDied","Data":"5df373756b3bc9d4f9f47a3ed2f163ed65be1e5360fd148b20b2c16595bbe8c1"}
Jul 17 05:30:29.220447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.220058    2582 scope.go:117] "RemoveContainer" containerID="2d681e02bf4e218d5971c3c3a5c33c81f8d0b66201fe73d9286fcb1adb33b4ae"
Jul 17 05:30:29.220447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.220347    2582 scope.go:117] "RemoveContainer" containerID="5df373756b3bc9d4f9f47a3ed2f163ed65be1e5360fd148b20b2c16595bbe8c1"
Jul 17 05:30:29.220613 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:29.220593    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-rs47p_odh-ai-gateway-infra(1b3c9aa6-8414-4848-9984-4a3f3742542c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c"
Jul 17 05:30:29.221415 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.221400    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"
Jul 17 05:30:29.221483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.221436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9" event={"ID":"d954536c-3bc5-46c9-9633-2a0ca7ee9366","Type":"ContainerDied","Data":"971b7663f22744d6dcaf04223ddd60fc51e8fb2aa49f2a0bd547e5ac267d4004"}
Jul 17 05:30:29.223225 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.223196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerStarted","Data":"43b8299994f6cd65143a16df23dae83b2b7cffc87185dd8bc144900cd35ebe85"}
Jul 17 05:30:29.231053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.231034    2582 scope.go:117] "RemoveContainer" containerID="111f97255d8269099f402c5438bee727dbf67747bd41afedf829b0d2cbe48c99"
Jul 17 05:30:29.246575 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.246550    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"]
Jul 17 05:30:29.252594 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.252570    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-ml7r9"]
Jul 17 05:30:29.255298 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:29.255277    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"]
Jul 17 05:30:30.230656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.230623    2582 generic.go:356] "Generic (PLEG): container finished" podID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerID="43b8299994f6cd65143a16df23dae83b2b7cffc87185dd8bc144900cd35ebe85" exitCode=7
Jul 17 05:30:30.231081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.230694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerDied","Data":"43b8299994f6cd65143a16df23dae83b2b7cffc87185dd8bc144900cd35ebe85"}
Jul 17 05:30:30.231081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.230757    2582 scope.go:117] "RemoveContainer" containerID="369d2a82da600996c3c7d46851b0be907c6cd8d9c2a04c43f37d220bb49af061"
Jul 17 05:30:30.231081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.231077    2582 scope.go:117] "RemoveContainer" containerID="43b8299994f6cd65143a16df23dae83b2b7cffc87185dd8bc144900cd35ebe85"
Jul 17 05:30:30.231356 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:30.231334    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2_odh-ai-gateway-infra(7df95432-6e9a-410e-96f4-85fa0b09d4e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7"
Jul 17 05:30:30.275914 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.275880    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"]
Jul 17 05:30:30.365689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.365663    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:30.407597 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.407571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9aa6-8414-4848-9984-4a3f3742542c-maas-api-service-ca\") pod \"1b3c9aa6-8414-4848-9984-4a3f3742542c\" (UID: \"1b3c9aa6-8414-4848-9984-4a3f3742542c\") "
Jul 17 05:30:30.407940 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.407919    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3c9aa6-8414-4848-9984-4a3f3742542c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b3c9aa6-8414-4848-9984-4a3f3742542c" (UID: "1b3c9aa6-8414-4848-9984-4a3f3742542c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:30:30.508708 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.508617    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3c9aa6-8414-4848-9984-4a3f3742542c-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:30:30.730952 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:30.730909    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" path="/var/lib/kubelet/pods/d954536c-3bc5-46c9-9633-2a0ca7ee9366/volumes"
Jul 17 05:30:31.239888 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.239829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p" event={"ID":"1b3c9aa6-8414-4848-9984-4a3f3742542c","Type":"ContainerDied","Data":"1519184a1a16a832add5482a77264d7f30e56a2dded12565fea7d69adca5a42d"}
Jul 17 05:30:31.239888 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.239861    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"
Jul 17 05:30:31.240359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.239897    2582 scope.go:117] "RemoveContainer" containerID="5df373756b3bc9d4f9f47a3ed2f163ed65be1e5360fd148b20b2c16595bbe8c1"
Jul 17 05:30:31.272313 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.272276    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"]
Jul 17 05:30:31.273552 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.273531    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-rs47p"]
Jul 17 05:30:31.373554 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.373528    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:31.418420 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.418391    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df95432-6e9a-410e-96f4-85fa0b09d4e7-maas-api-service-ca\") pod \"7df95432-6e9a-410e-96f4-85fa0b09d4e7\" (UID: \"7df95432-6e9a-410e-96f4-85fa0b09d4e7\") "
Jul 17 05:30:31.418730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.418710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df95432-6e9a-410e-96f4-85fa0b09d4e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7df95432-6e9a-410e-96f4-85fa0b09d4e7" (UID: "7df95432-6e9a-410e-96f4-85fa0b09d4e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:30:31.519470 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:31.519393    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7df95432-6e9a-410e-96f4-85fa0b09d4e7-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:30:32.245081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.245053    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"
Jul 17 05:30:32.245081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.245070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2" event={"ID":"7df95432-6e9a-410e-96f4-85fa0b09d4e7","Type":"ContainerDied","Data":"2bcd777b7843f00325970ee713af9c8bd97c09a42c620ff390aed97968f8f9e9"}
Jul 17 05:30:32.245638 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.245130    2582 scope.go:117] "RemoveContainer" containerID="43b8299994f6cd65143a16df23dae83b2b7cffc87185dd8bc144900cd35ebe85"
Jul 17 05:30:32.266169 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.266144    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"]
Jul 17 05:30:32.270421 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.270393    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-6gqc2"]
Jul 17 05:30:32.734810 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.734779    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" path="/var/lib/kubelet/pods/1b3c9aa6-8414-4848-9984-4a3f3742542c/volumes"
Jul 17 05:30:32.735100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:32.735087    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" path="/var/lib/kubelet/pods/7df95432-6e9a-410e-96f4-85fa0b09d4e7/volumes"
Jul 17 05:30:36.959371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"]
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959618    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959632    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959637    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959644    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959649    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959659    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959664    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959672    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959677    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959683    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959688    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959694    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959699    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959705    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959710    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959714    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959719    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959763    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:30:36.959759 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959770    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959777    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959782    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959788    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959792    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959797    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df95432-6e9a-410e-96f4-85fa0b09d4e7" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959802    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959807    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959813    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.960424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.959817    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:30:36.963970 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.963953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:36.966530 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.966511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:30:36.966618 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.966516    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:30:36.972232 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:36.972203    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"]
Jul 17 05:30:37.067697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:37.067659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3fe353-5273-4f1a-9a5b-dfb6f125759a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf\" (UID: \"da3fe353-5273-4f1a-9a5b-dfb6f125759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:37.168691 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:37.168650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3fe353-5273-4f1a-9a5b-dfb6f125759a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf\" (UID: \"da3fe353-5273-4f1a-9a5b-dfb6f125759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:37.169240 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:37.169222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3fe353-5273-4f1a-9a5b-dfb6f125759a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf\" (UID: \"da3fe353-5273-4f1a-9a5b-dfb6f125759a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:37.274746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:37.274667    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:37.432762 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:37.432724    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"]
Jul 17 05:30:37.435642 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:30:37.435615    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3fe353_5273_4f1a_9a5b_dfb6f125759a.slice/crio-7aa2a7bc7428727acb20563eccc35d28549947371daa9b191fdce63d6a5b0bfb WatchSource:0}: Error finding container 7aa2a7bc7428727acb20563eccc35d28549947371daa9b191fdce63d6a5b0bfb: Status 404 returned error can't find the container with id 7aa2a7bc7428727acb20563eccc35d28549947371daa9b191fdce63d6a5b0bfb
Jul 17 05:30:38.279500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:38.279409    2582 generic.go:356] "Generic (PLEG): container finished" podID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerID="4105fa699ff7060ee7120464cb3c7ae18055480282833addeb2ace9d6a2dda3f" exitCode=6
Jul 17 05:30:38.279500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:38.279488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" event={"ID":"da3fe353-5273-4f1a-9a5b-dfb6f125759a","Type":"ContainerDied","Data":"4105fa699ff7060ee7120464cb3c7ae18055480282833addeb2ace9d6a2dda3f"}
Jul 17 05:30:38.279995 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:38.279563    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" event={"ID":"da3fe353-5273-4f1a-9a5b-dfb6f125759a","Type":"ContainerStarted","Data":"7aa2a7bc7428727acb20563eccc35d28549947371daa9b191fdce63d6a5b0bfb"}
Jul 17 05:30:38.279995 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:38.279651    2582 scope.go:117] "RemoveContainer" containerID="4105fa699ff7060ee7120464cb3c7ae18055480282833addeb2ace9d6a2dda3f"
Jul 17 05:30:39.286829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:39.286799    2582 generic.go:356] "Generic (PLEG): container finished" podID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerID="0aec34237d05ac8d5ea1fab7c5516b69d270a727bfb45c28d535096dea37630e" exitCode=6
Jul 17 05:30:39.287224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:39.286886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" event={"ID":"da3fe353-5273-4f1a-9a5b-dfb6f125759a","Type":"ContainerDied","Data":"0aec34237d05ac8d5ea1fab7c5516b69d270a727bfb45c28d535096dea37630e"}
Jul 17 05:30:39.287224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:39.286943    2582 scope.go:117] "RemoveContainer" containerID="4105fa699ff7060ee7120464cb3c7ae18055480282833addeb2ace9d6a2dda3f"
Jul 17 05:30:39.287224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:39.287047    2582 scope.go:117] "RemoveContainer" containerID="0aec34237d05ac8d5ea1fab7c5516b69d270a727bfb45c28d535096dea37630e"
Jul 17 05:30:39.287402 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:39.287288    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf_odh-ai-gateway-infra(da3fe353-5273-4f1a-9a5b-dfb6f125759a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a"
Jul 17 05:30:40.292785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:40.292758    2582 scope.go:117] "RemoveContainer" containerID="0aec34237d05ac8d5ea1fab7c5516b69d270a727bfb45c28d535096dea37630e"
Jul 17 05:30:40.293193 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:30:40.292928    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf_odh-ai-gateway-infra(da3fe353-5273-4f1a-9a5b-dfb6f125759a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a"
Jul 17 05:30:55.724772 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:55.724731    2582 scope.go:117] "RemoveContainer" containerID="0aec34237d05ac8d5ea1fab7c5516b69d270a727bfb45c28d535096dea37630e"
Jul 17 05:30:56.368702 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:56.368605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" event={"ID":"da3fe353-5273-4f1a-9a5b-dfb6f125759a","Type":"ContainerStarted","Data":"acff6931b33f643523efb3bf4aa6bced08bba33a182ebb651f0f0b76ca006567"}
Jul 17 05:30:56.386555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:56.386503    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" podStartSLOduration=20.386474788 podStartE2EDuration="20.386474788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:30:56.3844419 +0000 UTC m=+2460.121504239" watchObservedRunningTime="2026-07-17 05:30:56.386474788 +0000 UTC m=+2460.123537124"
Jul 17 05:30:57.270493 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.270455    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"]
Jul 17 05:30:57.377054 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.376955    2582 generic.go:356] "Generic (PLEG): container finished" podID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerID="acff6931b33f643523efb3bf4aa6bced08bba33a182ebb651f0f0b76ca006567" exitCode=6
Jul 17 05:30:57.377054 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.377045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" event={"ID":"da3fe353-5273-4f1a-9a5b-dfb6f125759a","Type":"ContainerDied","Data":"acff6931b33f643523efb3bf4aa6bced08bba33a182ebb651f0f0b76ca006567"}
Jul 17 05:30:57.377317 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.377104    2582 scope.go:117] "RemoveContainer" containerID="0aec34237d05ac8d5ea1fab7c5516b69d270a727bfb45c28d535096dea37630e"
Jul 17 05:30:57.503808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.503786    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:57.651128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.651035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3fe353-5273-4f1a-9a5b-dfb6f125759a-maas-api-service-ca\") pod \"da3fe353-5273-4f1a-9a5b-dfb6f125759a\" (UID: \"da3fe353-5273-4f1a-9a5b-dfb6f125759a\") "
Jul 17 05:30:57.651483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.651451    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3fe353-5273-4f1a-9a5b-dfb6f125759a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da3fe353-5273-4f1a-9a5b-dfb6f125759a" (UID: "da3fe353-5273-4f1a-9a5b-dfb6f125759a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:30:57.751507 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:57.751474    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3fe353-5273-4f1a-9a5b-dfb6f125759a-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:30:58.383022 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:58.382994    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"
Jul 17 05:30:58.383022 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:58.383020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf" event={"ID":"da3fe353-5273-4f1a-9a5b-dfb6f125759a","Type":"ContainerDied","Data":"7aa2a7bc7428727acb20563eccc35d28549947371daa9b191fdce63d6a5b0bfb"}
Jul 17 05:30:58.383544 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:58.383080    2582 scope.go:117] "RemoveContainer" containerID="acff6931b33f643523efb3bf4aa6bced08bba33a182ebb651f0f0b76ca006567"
Jul 17 05:30:58.402955 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:58.402927    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"]
Jul 17 05:30:58.405045 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:58.405022    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqjf"]
Jul 17 05:30:58.729595 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:30:58.729562    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" path="/var/lib/kubelet/pods/da3fe353-5273-4f1a-9a5b-dfb6f125759a/volumes"
Jul 17 05:31:08.902595 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.902554    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"]
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.902966    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.902983    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.902995    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903002    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b3c9aa6-8414-4848-9984-4a3f3742542c" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903013    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33058111-4c95-481d-bf2c-69dc079f2644" containerName="cleanup"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903033    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903041    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903069    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903076    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:31:08.903154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903154    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.903738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903165    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d954536c-3bc5-46c9-9633-2a0ca7ee9366" containerName="revoke-keys"
Jul 17 05:31:08.903738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903174    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.903738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.903183    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:31:08.906328 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.906306    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:08.908914 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.908893    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:31:08.909041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.908968    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:31:08.913798 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.913707    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"]
Jul 17 05:31:08.933512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:08.933475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc7aab7-9142-4c82-9934-fcd4d4f3429c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr\" (UID: \"edc7aab7-9142-4c82-9934-fcd4d4f3429c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:09.034398 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:09.034362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc7aab7-9142-4c82-9934-fcd4d4f3429c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr\" (UID: \"edc7aab7-9142-4c82-9934-fcd4d4f3429c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:09.034978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:09.034961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc7aab7-9142-4c82-9934-fcd4d4f3429c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr\" (UID: \"edc7aab7-9142-4c82-9934-fcd4d4f3429c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:09.217359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:09.217309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:09.355248 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:09.355210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"]
Jul 17 05:31:09.358365 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:31:09.358329    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc7aab7_9142_4c82_9934_fcd4d4f3429c.slice/crio-82e6ae274e5f82b8d07d4473a6ac3c6f7f486c59c1b8f05778dbddd8abb836a9 WatchSource:0}: Error finding container 82e6ae274e5f82b8d07d4473a6ac3c6f7f486c59c1b8f05778dbddd8abb836a9: Status 404 returned error can't find the container with id 82e6ae274e5f82b8d07d4473a6ac3c6f7f486c59c1b8f05778dbddd8abb836a9
Jul 17 05:31:09.434650 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:09.434617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" event={"ID":"edc7aab7-9142-4c82-9934-fcd4d4f3429c","Type":"ContainerStarted","Data":"82e6ae274e5f82b8d07d4473a6ac3c6f7f486c59c1b8f05778dbddd8abb836a9"}
Jul 17 05:31:10.440305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:10.440249    2582 generic.go:356] "Generic (PLEG): container finished" podID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerID="f57e21dba03d81cc20e511ec9aed9db312d89f2ecf2b9b98d54d65a247a71016" exitCode=6
Jul 17 05:31:10.440800 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:10.440369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" event={"ID":"edc7aab7-9142-4c82-9934-fcd4d4f3429c","Type":"ContainerDied","Data":"f57e21dba03d81cc20e511ec9aed9db312d89f2ecf2b9b98d54d65a247a71016"}
Jul 17 05:31:10.440800 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:10.440591    2582 scope.go:117] "RemoveContainer" containerID="f57e21dba03d81cc20e511ec9aed9db312d89f2ecf2b9b98d54d65a247a71016"
Jul 17 05:31:11.447198 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:11.447165    2582 generic.go:356] "Generic (PLEG): container finished" podID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerID="5471443bd62bea633521ecdf672e4c7c657bc886be6cf26c528903f6af53a888" exitCode=6
Jul 17 05:31:11.447681 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:11.447224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" event={"ID":"edc7aab7-9142-4c82-9934-fcd4d4f3429c","Type":"ContainerDied","Data":"5471443bd62bea633521ecdf672e4c7c657bc886be6cf26c528903f6af53a888"}
Jul 17 05:31:11.447681 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:11.447294    2582 scope.go:117] "RemoveContainer" containerID="f57e21dba03d81cc20e511ec9aed9db312d89f2ecf2b9b98d54d65a247a71016"
Jul 17 05:31:11.447681 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:11.447441    2582 scope.go:117] "RemoveContainer" containerID="5471443bd62bea633521ecdf672e4c7c657bc886be6cf26c528903f6af53a888"
Jul 17 05:31:11.447681 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:31:11.447649    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr_odh-ai-gateway-infra(edc7aab7-9142-4c82-9934-fcd4d4f3429c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c"
Jul 17 05:31:12.453277 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:12.453241    2582 scope.go:117] "RemoveContainer" containerID="5471443bd62bea633521ecdf672e4c7c657bc886be6cf26c528903f6af53a888"
Jul 17 05:31:12.453730 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:31:12.453494    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr_odh-ai-gateway-infra(edc7aab7-9142-4c82-9934-fcd4d4f3429c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c"
Jul 17 05:31:24.729656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:24.729629    2582 scope.go:117] "RemoveContainer" containerID="5471443bd62bea633521ecdf672e4c7c657bc886be6cf26c528903f6af53a888"
Jul 17 05:31:25.509236 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:25.509202    2582 generic.go:356] "Generic (PLEG): container finished" podID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerID="2235e27c768faa8804b8f74b9db84265a006b9ea317c995261183ed9e2be2416" exitCode=6
Jul 17 05:31:25.509402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:25.509278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" event={"ID":"edc7aab7-9142-4c82-9934-fcd4d4f3429c","Type":"ContainerDied","Data":"2235e27c768faa8804b8f74b9db84265a006b9ea317c995261183ed9e2be2416"}
Jul 17 05:31:25.509402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:25.509325    2582 scope.go:117] "RemoveContainer" containerID="5471443bd62bea633521ecdf672e4c7c657bc886be6cf26c528903f6af53a888"
Jul 17 05:31:25.509606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:25.509586    2582 scope.go:117] "RemoveContainer" containerID="2235e27c768faa8804b8f74b9db84265a006b9ea317c995261183ed9e2be2416"
Jul 17 05:31:25.509819 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:31:25.509794    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr_odh-ai-gateway-infra(edc7aab7-9142-4c82-9934-fcd4d4f3429c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c"
Jul 17 05:31:26.535651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:26.535610    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"]
Jul 17 05:31:26.666696 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:26.666667    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:26.776835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:26.776799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc7aab7-9142-4c82-9934-fcd4d4f3429c-maas-api-service-ca\") pod \"edc7aab7-9142-4c82-9934-fcd4d4f3429c\" (UID: \"edc7aab7-9142-4c82-9934-fcd4d4f3429c\") "
Jul 17 05:31:26.777143 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:26.777122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc7aab7-9142-4c82-9934-fcd4d4f3429c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edc7aab7-9142-4c82-9934-fcd4d4f3429c" (UID: "edc7aab7-9142-4c82-9934-fcd4d4f3429c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:31:26.877886 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:26.877805    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc7aab7-9142-4c82-9934-fcd4d4f3429c-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:31:27.519921 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:27.519887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr" event={"ID":"edc7aab7-9142-4c82-9934-fcd4d4f3429c","Type":"ContainerDied","Data":"82e6ae274e5f82b8d07d4473a6ac3c6f7f486c59c1b8f05778dbddd8abb836a9"}
Jul 17 05:31:27.519921 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:27.519918    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"
Jul 17 05:31:27.520178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:27.519947    2582 scope.go:117] "RemoveContainer" containerID="2235e27c768faa8804b8f74b9db84265a006b9ea317c995261183ed9e2be2416"
Jul 17 05:31:27.540422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:27.540391    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"]
Jul 17 05:31:27.547721 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:27.547694    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sptmr"]
Jul 17 05:31:28.730621 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:31:28.730580    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" path="/var/lib/kubelet/pods/edc7aab7-9142-4c82-9934-fcd4d4f3429c/volumes"
Jul 17 05:32:31.503409 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503377    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"]
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503657    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503668    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503676    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503682    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503700    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503705    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3fe353-5273-4f1a-9a5b-dfb6f125759a" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503749    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503756    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.503927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.503762    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:31.506603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.506588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:31.509684 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.509657    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:32:31.509786 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.509680    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:32:31.514898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.514873    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"]
Jul 17 05:32:31.617029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.616995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eed1d48-d972-4463-b08b-9db904e1c353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-857d512b-mvdd5\" (UID: \"1eed1d48-d972-4463-b08b-9db904e1c353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:31.718339 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.718297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eed1d48-d972-4463-b08b-9db904e1c353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-857d512b-mvdd5\" (UID: \"1eed1d48-d972-4463-b08b-9db904e1c353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:31.718939 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.718916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eed1d48-d972-4463-b08b-9db904e1c353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-857d512b-mvdd5\" (UID: \"1eed1d48-d972-4463-b08b-9db904e1c353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:31.817355 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.817274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:31.948624 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:31.948589    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"]
Jul 17 05:32:31.951207 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:32:31.951182    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eed1d48_d972_4463_b08b_9db904e1c353.slice/crio-3991aed8f508868f11de54c09ac9efc393c63f198698e03076fc98c3a4cec48b WatchSource:0}: Error finding container 3991aed8f508868f11de54c09ac9efc393c63f198698e03076fc98c3a4cec48b: Status 404 returned error can't find the container with id 3991aed8f508868f11de54c09ac9efc393c63f198698e03076fc98c3a4cec48b
Jul 17 05:32:32.790835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:32.790801    2582 generic.go:356] "Generic (PLEG): container finished" podID="1eed1d48-d972-4463-b08b-9db904e1c353" containerID="aaf7ba062ba5e0067d740eaa8897fadb0d86bce034f9de2c014edb2e273e2dbd" exitCode=0
Jul 17 05:32:32.791286 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:32.790889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5" event={"ID":"1eed1d48-d972-4463-b08b-9db904e1c353","Type":"ContainerDied","Data":"aaf7ba062ba5e0067d740eaa8897fadb0d86bce034f9de2c014edb2e273e2dbd"}
Jul 17 05:32:32.791286 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:32.790937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5" event={"ID":"1eed1d48-d972-4463-b08b-9db904e1c353","Type":"ContainerStarted","Data":"3991aed8f508868f11de54c09ac9efc393c63f198698e03076fc98c3a4cec48b"}
Jul 17 05:32:33.923675 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:33.923653    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:34.038162 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:34.038124    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eed1d48-d972-4463-b08b-9db904e1c353-maas-api-service-ca\") pod \"1eed1d48-d972-4463-b08b-9db904e1c353\" (UID: \"1eed1d48-d972-4463-b08b-9db904e1c353\") "
Jul 17 05:32:34.038549 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:34.038530    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eed1d48-d972-4463-b08b-9db904e1c353-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1eed1d48-d972-4463-b08b-9db904e1c353" (UID: "1eed1d48-d972-4463-b08b-9db904e1c353"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:32:34.139099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:34.139008    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eed1d48-d972-4463-b08b-9db904e1c353-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:32:34.800270 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:34.800226    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5"
Jul 17 05:32:34.800467 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:34.800225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-857d512b-mvdd5" event={"ID":"1eed1d48-d972-4463-b08b-9db904e1c353","Type":"ContainerDied","Data":"3991aed8f508868f11de54c09ac9efc393c63f198698e03076fc98c3a4cec48b"}
Jul 17 05:32:34.800467 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:34.800351    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3991aed8f508868f11de54c09ac9efc393c63f198698e03076fc98c3a4cec48b"
Jul 17 05:32:37.057567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.057533    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"]
Jul 17 05:32:37.057943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.057830    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eed1d48-d972-4463-b08b-9db904e1c353" containerName="revoke-keys"
Jul 17 05:32:37.057943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.057844    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eed1d48-d972-4463-b08b-9db904e1c353" containerName="revoke-keys"
Jul 17 05:32:37.057943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.057861    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:37.057943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.057867    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc7aab7-9142-4c82-9934-fcd4d4f3429c" containerName="revoke-keys"
Jul 17 05:32:37.057943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.057918    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eed1d48-d972-4463-b08b-9db904e1c353" containerName="revoke-keys"
Jul 17 05:32:37.060841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.060825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:37.063720 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.063700    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:32:37.063808 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.063752    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:32:37.069527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.069503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"]
Jul 17 05:32:37.163970 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.163930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6251c7b-4922-452f-bb2d-375a26517277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml\" (UID: \"d6251c7b-4922-452f-bb2d-375a26517277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:37.264844 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.264812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6251c7b-4922-452f-bb2d-375a26517277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml\" (UID: \"d6251c7b-4922-452f-bb2d-375a26517277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:37.265493 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.265474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6251c7b-4922-452f-bb2d-375a26517277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml\" (UID: \"d6251c7b-4922-452f-bb2d-375a26517277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:37.371720 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.371640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:37.501014 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.500989    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"]
Jul 17 05:32:37.503644 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:32:37.503621    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6251c7b_4922_452f_bb2d_375a26517277.slice/crio-f338768150fa46070e05d36742a04a3bd46c70c87a970d330d0dc417ba9d2b86 WatchSource:0}: Error finding container f338768150fa46070e05d36742a04a3bd46c70c87a970d330d0dc417ba9d2b86: Status 404 returned error can't find the container with id f338768150fa46070e05d36742a04a3bd46c70c87a970d330d0dc417ba9d2b86
Jul 17 05:32:37.813296 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.813245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" event={"ID":"d6251c7b-4922-452f-bb2d-375a26517277","Type":"ContainerStarted","Data":"0ec4e16ad5fa0586e5e30ba781929e085628b8c7e1a547b08918577526abde40"}
Jul 17 05:32:37.813402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.813316    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" event={"ID":"d6251c7b-4922-452f-bb2d-375a26517277","Type":"ContainerStarted","Data":"f338768150fa46070e05d36742a04a3bd46c70c87a970d330d0dc417ba9d2b86"}
Jul 17 05:32:37.828617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:37.828560    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" podStartSLOduration=0.82854426 podStartE2EDuration="828.54426ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:32:37.827965329 +0000 UTC m=+2561.565027666" watchObservedRunningTime="2026-07-17 05:32:37.82854426 +0000 UTC m=+2561.565606651"
Jul 17 05:32:38.818299 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:38.818246    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6251c7b-4922-452f-bb2d-375a26517277" containerID="0ec4e16ad5fa0586e5e30ba781929e085628b8c7e1a547b08918577526abde40" exitCode=6
Jul 17 05:32:38.818709 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:38.818337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" event={"ID":"d6251c7b-4922-452f-bb2d-375a26517277","Type":"ContainerDied","Data":"0ec4e16ad5fa0586e5e30ba781929e085628b8c7e1a547b08918577526abde40"}
Jul 17 05:32:38.818709 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:38.818554    2582 scope.go:117] "RemoveContainer" containerID="0ec4e16ad5fa0586e5e30ba781929e085628b8c7e1a547b08918577526abde40"
Jul 17 05:32:39.824532 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:39.824495    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6251c7b-4922-452f-bb2d-375a26517277" containerID="24696d7db49f0a510c206c2643ffde6505c2d0570781717267d8a467b6512788" exitCode=6
Jul 17 05:32:39.825016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:39.824557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" event={"ID":"d6251c7b-4922-452f-bb2d-375a26517277","Type":"ContainerDied","Data":"24696d7db49f0a510c206c2643ffde6505c2d0570781717267d8a467b6512788"}
Jul 17 05:32:39.825016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:39.824631    2582 scope.go:117] "RemoveContainer" containerID="0ec4e16ad5fa0586e5e30ba781929e085628b8c7e1a547b08918577526abde40"
Jul 17 05:32:39.825016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:39.824798    2582 scope.go:117] "RemoveContainer" containerID="24696d7db49f0a510c206c2643ffde6505c2d0570781717267d8a467b6512788"
Jul 17 05:32:39.825184 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:39.825037    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml_odh-ai-gateway-infra(d6251c7b-4922-452f-bb2d-375a26517277)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" podUID="d6251c7b-4922-452f-bb2d-375a26517277"
Jul 17 05:32:40.830412 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:40.830383    2582 scope.go:117] "RemoveContainer" containerID="24696d7db49f0a510c206c2643ffde6505c2d0570781717267d8a467b6512788"
Jul 17 05:32:40.830768 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:40.830556    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml_odh-ai-gateway-infra(d6251c7b-4922-452f-bb2d-375a26517277)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" podUID="d6251c7b-4922-452f-bb2d-375a26517277"
Jul 17 05:32:48.998314 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:48.998276    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"]
Jul 17 05:32:49.002552 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.002527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:32:49.009670 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.009639    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"]
Jul 17 05:32:49.164307 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.164271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f811138a-14cb-4c16-a12a-c4c2ef399dc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9\" (UID: \"f811138a-14cb-4c16-a12a-c4c2ef399dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:32:49.265536 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.265432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f811138a-14cb-4c16-a12a-c4c2ef399dc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9\" (UID: \"f811138a-14cb-4c16-a12a-c4c2ef399dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:32:49.266066 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.266045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f811138a-14cb-4c16-a12a-c4c2ef399dc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9\" (UID: \"f811138a-14cb-4c16-a12a-c4c2ef399dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:32:49.313784 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.313744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:32:49.442160 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.441966    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"]
Jul 17 05:32:49.444488 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:32:49.444458    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf811138a_14cb_4c16_a12a_c4c2ef399dc7.slice/crio-3275c56bcd8bd244e63e3b8d396603c74980cc9dec4df2fbdf4764069d788938 WatchSource:0}: Error finding container 3275c56bcd8bd244e63e3b8d396603c74980cc9dec4df2fbdf4764069d788938: Status 404 returned error can't find the container with id 3275c56bcd8bd244e63e3b8d396603c74980cc9dec4df2fbdf4764069d788938
Jul 17 05:32:49.872155 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.872120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" event={"ID":"f811138a-14cb-4c16-a12a-c4c2ef399dc7","Type":"ContainerStarted","Data":"d44cfb96ea2a53bb81ad42d0ed8042bc76b45aaa496a620363ba9bbac0573a8f"}
Jul 17 05:32:49.872377 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.872173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" event={"ID":"f811138a-14cb-4c16-a12a-c4c2ef399dc7","Type":"ContainerStarted","Data":"3275c56bcd8bd244e63e3b8d396603c74980cc9dec4df2fbdf4764069d788938"}
Jul 17 05:32:49.889039 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:49.888989    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" podStartSLOduration=1.888975591 podStartE2EDuration="1.888975591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:32:49.888678928 +0000 UTC m=+2573.625741275" watchObservedRunningTime="2026-07-17 05:32:49.888975591 +0000 UTC m=+2573.626037923"
Jul 17 05:32:50.878768 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:50.878736    2582 generic.go:356] "Generic (PLEG): container finished" podID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerID="d44cfb96ea2a53bb81ad42d0ed8042bc76b45aaa496a620363ba9bbac0573a8f" exitCode=6
Jul 17 05:32:50.879147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:50.878800    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" event={"ID":"f811138a-14cb-4c16-a12a-c4c2ef399dc7","Type":"ContainerDied","Data":"d44cfb96ea2a53bb81ad42d0ed8042bc76b45aaa496a620363ba9bbac0573a8f"}
Jul 17 05:32:50.879147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:50.878979    2582 scope.go:117] "RemoveContainer" containerID="d44cfb96ea2a53bb81ad42d0ed8042bc76b45aaa496a620363ba9bbac0573a8f"
Jul 17 05:32:51.890212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:51.890127    2582 generic.go:356] "Generic (PLEG): container finished" podID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerID="5ee1e236559f57e00566ff47d0c90567d7b371e3eb42848bd8b1fb667f6be3ec" exitCode=6
Jul 17 05:32:51.890212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:51.890163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" event={"ID":"f811138a-14cb-4c16-a12a-c4c2ef399dc7","Type":"ContainerDied","Data":"5ee1e236559f57e00566ff47d0c90567d7b371e3eb42848bd8b1fb667f6be3ec"}
Jul 17 05:32:51.890629 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:51.890227    2582 scope.go:117] "RemoveContainer" containerID="d44cfb96ea2a53bb81ad42d0ed8042bc76b45aaa496a620363ba9bbac0573a8f"
Jul 17 05:32:51.890629 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:51.890392    2582 scope.go:117] "RemoveContainer" containerID="5ee1e236559f57e00566ff47d0c90567d7b371e3eb42848bd8b1fb667f6be3ec"
Jul 17 05:32:51.890629 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:51.890613    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9_odh-ai-gateway-infra(f811138a-14cb-4c16-a12a-c4c2ef399dc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7"
Jul 17 05:32:52.725794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:52.725760    2582 scope.go:117] "RemoveContainer" containerID="24696d7db49f0a510c206c2643ffde6505c2d0570781717267d8a467b6512788"
Jul 17 05:32:52.895943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:52.895913    2582 scope.go:117] "RemoveContainer" containerID="5ee1e236559f57e00566ff47d0c90567d7b371e3eb42848bd8b1fb667f6be3ec"
Jul 17 05:32:52.896348 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:52.896125    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9_odh-ai-gateway-infra(f811138a-14cb-4c16-a12a-c4c2ef399dc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7"
Jul 17 05:32:53.901887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:53.901853    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6251c7b-4922-452f-bb2d-375a26517277" containerID="c9306a748b61a1457f730d96ab6c1d7524b4abe5dc42c05c54376eb522d55b00" exitCode=6
Jul 17 05:32:53.901887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:53.901890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" event={"ID":"d6251c7b-4922-452f-bb2d-375a26517277","Type":"ContainerDied","Data":"c9306a748b61a1457f730d96ab6c1d7524b4abe5dc42c05c54376eb522d55b00"}
Jul 17 05:32:53.902395 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:53.901965    2582 scope.go:117] "RemoveContainer" containerID="24696d7db49f0a510c206c2643ffde6505c2d0570781717267d8a467b6512788"
Jul 17 05:32:53.902395 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:53.902252    2582 scope.go:117] "RemoveContainer" containerID="c9306a748b61a1457f730d96ab6c1d7524b4abe5dc42c05c54376eb522d55b00"
Jul 17 05:32:53.902526 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:53.902500    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml_odh-ai-gateway-infra(d6251c7b-4922-452f-bb2d-375a26517277)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" podUID="d6251c7b-4922-452f-bb2d-375a26517277"
Jul 17 05:32:54.438309 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.438247    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"]
Jul 17 05:32:54.442617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.442599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:32:54.449004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.448941    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"]
Jul 17 05:32:54.610403 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.610342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96bcef7-447e-4903-a512-8f7d7bdd5c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg\" (UID: \"f96bcef7-447e-4903-a512-8f7d7bdd5c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:32:54.711720 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.711689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96bcef7-447e-4903-a512-8f7d7bdd5c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg\" (UID: \"f96bcef7-447e-4903-a512-8f7d7bdd5c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:32:54.712331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.712312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96bcef7-447e-4903-a512-8f7d7bdd5c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg\" (UID: \"f96bcef7-447e-4903-a512-8f7d7bdd5c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:32:54.754462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.754431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:32:54.884185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.884152    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"]
Jul 17 05:32:54.887429 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:32:54.887401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96bcef7_447e_4903_a512_8f7d7bdd5c64.slice/crio-29877d29867cb5d0d8f9955a3b5f1bdbb8bac583e07d91cedac4b9b215168352 WatchSource:0}: Error finding container 29877d29867cb5d0d8f9955a3b5f1bdbb8bac583e07d91cedac4b9b215168352: Status 404 returned error can't find the container with id 29877d29867cb5d0d8f9955a3b5f1bdbb8bac583e07d91cedac4b9b215168352
Jul 17 05:32:54.910976 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.910931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" event={"ID":"f96bcef7-447e-4903-a512-8f7d7bdd5c64","Type":"ContainerStarted","Data":"29877d29867cb5d0d8f9955a3b5f1bdbb8bac583e07d91cedac4b9b215168352"}
Jul 17 05:32:54.926557 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:54.926533    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"]
Jul 17 05:32:55.041964 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.041938    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:55.215348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.215325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6251c7b-4922-452f-bb2d-375a26517277-maas-api-service-ca\") pod \"d6251c7b-4922-452f-bb2d-375a26517277\" (UID: \"d6251c7b-4922-452f-bb2d-375a26517277\") "
Jul 17 05:32:55.215728 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.215705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6251c7b-4922-452f-bb2d-375a26517277-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6251c7b-4922-452f-bb2d-375a26517277" (UID: "d6251c7b-4922-452f-bb2d-375a26517277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:32:55.316184 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.316097    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6251c7b-4922-452f-bb2d-375a26517277-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:32:55.920840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.920745    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"
Jul 17 05:32:55.920840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.920752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml" event={"ID":"d6251c7b-4922-452f-bb2d-375a26517277","Type":"ContainerDied","Data":"f338768150fa46070e05d36742a04a3bd46c70c87a970d330d0dc417ba9d2b86"}
Jul 17 05:32:55.921412 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.920905    2582 scope.go:117] "RemoveContainer" containerID="c9306a748b61a1457f730d96ab6c1d7524b4abe5dc42c05c54376eb522d55b00"
Jul 17 05:32:55.922376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.922349    2582 generic.go:356] "Generic (PLEG): container finished" podID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerID="6761865eef0542f95ef12c9c8a26e38f5014e481a7827faf98c177982b2ff542" exitCode=6
Jul 17 05:32:55.922487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.922416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" event={"ID":"f96bcef7-447e-4903-a512-8f7d7bdd5c64","Type":"ContainerDied","Data":"6761865eef0542f95ef12c9c8a26e38f5014e481a7827faf98c177982b2ff542"}
Jul 17 05:32:55.922587 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.922575    2582 scope.go:117] "RemoveContainer" containerID="6761865eef0542f95ef12c9c8a26e38f5014e481a7827faf98c177982b2ff542"
Jul 17 05:32:55.953318 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.953294    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"]
Jul 17 05:32:55.957249 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:55.957229    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-kmqml"]
Jul 17 05:32:56.730563 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:56.730527    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6251c7b-4922-452f-bb2d-375a26517277" path="/var/lib/kubelet/pods/d6251c7b-4922-452f-bb2d-375a26517277/volumes"
Jul 17 05:32:56.933819 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:56.933777    2582 generic.go:356] "Generic (PLEG): container finished" podID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerID="0e5820571dc4cd2a620e88c8b7848f62d8ca5a5e4db1a10b11bc2ee2dd903f2a" exitCode=6
Jul 17 05:32:56.934216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:56.933858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" event={"ID":"f96bcef7-447e-4903-a512-8f7d7bdd5c64","Type":"ContainerDied","Data":"0e5820571dc4cd2a620e88c8b7848f62d8ca5a5e4db1a10b11bc2ee2dd903f2a"}
Jul 17 05:32:56.934216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:56.933949    2582 scope.go:117] "RemoveContainer" containerID="6761865eef0542f95ef12c9c8a26e38f5014e481a7827faf98c177982b2ff542"
Jul 17 05:32:56.934216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:56.934100    2582 scope.go:117] "RemoveContainer" containerID="0e5820571dc4cd2a620e88c8b7848f62d8ca5a5e4db1a10b11bc2ee2dd903f2a"
Jul 17 05:32:56.934407 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:56.934384    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg_odh-ai-gateway-infra(f96bcef7-447e-4903-a512-8f7d7bdd5c64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64"
Jul 17 05:32:57.941053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:32:57.941021    2582 scope.go:117] "RemoveContainer" containerID="0e5820571dc4cd2a620e88c8b7848f62d8ca5a5e4db1a10b11bc2ee2dd903f2a"
Jul 17 05:32:57.941462 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:32:57.941251    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg_odh-ai-gateway-infra(f96bcef7-447e-4903-a512-8f7d7bdd5c64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64"
Jul 17 05:33:04.725752 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:04.725719    2582 scope.go:117] "RemoveContainer" containerID="5ee1e236559f57e00566ff47d0c90567d7b371e3eb42848bd8b1fb667f6be3ec"
Jul 17 05:33:05.976461 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:05.976428    2582 generic.go:356] "Generic (PLEG): container finished" podID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerID="01ad31ad59af295827e9c0e2b06979bee41166fe8a3353bf4ab84278ddbe1163" exitCode=6
Jul 17 05:33:05.976938 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:05.976486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" event={"ID":"f811138a-14cb-4c16-a12a-c4c2ef399dc7","Type":"ContainerDied","Data":"01ad31ad59af295827e9c0e2b06979bee41166fe8a3353bf4ab84278ddbe1163"}
Jul 17 05:33:05.976938 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:05.976545    2582 scope.go:117] "RemoveContainer" containerID="5ee1e236559f57e00566ff47d0c90567d7b371e3eb42848bd8b1fb667f6be3ec"
Jul 17 05:33:05.976938 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:05.976915    2582 scope.go:117] "RemoveContainer" containerID="01ad31ad59af295827e9c0e2b06979bee41166fe8a3353bf4ab84278ddbe1163"
Jul 17 05:33:05.977134 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:05.977115    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9_odh-ai-gateway-infra(f811138a-14cb-4c16-a12a-c4c2ef399dc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7"
Jul 17 05:33:07.002594 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.002564    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"]
Jul 17 05:33:07.135075 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.135053    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:33:07.217111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.217082    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f811138a-14cb-4c16-a12a-c4c2ef399dc7-maas-api-service-ca\") pod \"f811138a-14cb-4c16-a12a-c4c2ef399dc7\" (UID: \"f811138a-14cb-4c16-a12a-c4c2ef399dc7\") "
Jul 17 05:33:07.217472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.217450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f811138a-14cb-4c16-a12a-c4c2ef399dc7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f811138a-14cb-4c16-a12a-c4c2ef399dc7" (UID: "f811138a-14cb-4c16-a12a-c4c2ef399dc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:33:07.317597 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.317510    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f811138a-14cb-4c16-a12a-c4c2ef399dc7-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:33:07.987544 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.987513    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"
Jul 17 05:33:07.987730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.987542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9" event={"ID":"f811138a-14cb-4c16-a12a-c4c2ef399dc7","Type":"ContainerDied","Data":"3275c56bcd8bd244e63e3b8d396603c74980cc9dec4df2fbdf4764069d788938"}
Jul 17 05:33:07.987730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:07.987611    2582 scope.go:117] "RemoveContainer" containerID="01ad31ad59af295827e9c0e2b06979bee41166fe8a3353bf4ab84278ddbe1163"
Jul 17 05:33:08.008739 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:08.008713    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"]
Jul 17 05:33:08.017288 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:08.017250    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-4wkj9"]
Jul 17 05:33:08.729548 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:08.729520    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" path="/var/lib/kubelet/pods/f811138a-14cb-4c16-a12a-c4c2ef399dc7/volumes"
Jul 17 05:33:11.725007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:11.724975    2582 scope.go:117] "RemoveContainer" containerID="0e5820571dc4cd2a620e88c8b7848f62d8ca5a5e4db1a10b11bc2ee2dd903f2a"
Jul 17 05:33:12.006070 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:12.005984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" event={"ID":"f96bcef7-447e-4903-a512-8f7d7bdd5c64","Type":"ContainerStarted","Data":"c52836e56dfb10f66de22053e8789dd0e0d0d2473b16b86f3198b629c211d4a6"}
Jul 17 05:33:12.022277 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:12.022209    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" podStartSLOduration=18.022194095 podStartE2EDuration="18.022194095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:33:12.020781553 +0000 UTC m=+2595.757843888" watchObservedRunningTime="2026-07-17 05:33:12.022194095 +0000 UTC m=+2595.759256430"
Jul 17 05:33:13.012466 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:13.012434    2582 generic.go:356] "Generic (PLEG): container finished" podID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerID="c52836e56dfb10f66de22053e8789dd0e0d0d2473b16b86f3198b629c211d4a6" exitCode=6
Jul 17 05:33:13.012897 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:13.012488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" event={"ID":"f96bcef7-447e-4903-a512-8f7d7bdd5c64","Type":"ContainerDied","Data":"c52836e56dfb10f66de22053e8789dd0e0d0d2473b16b86f3198b629c211d4a6"}
Jul 17 05:33:13.012897 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:13.012542    2582 scope.go:117] "RemoveContainer" containerID="0e5820571dc4cd2a620e88c8b7848f62d8ca5a5e4db1a10b11bc2ee2dd903f2a"
Jul 17 05:33:13.012897 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:13.012883    2582 scope.go:117] "RemoveContainer" containerID="c52836e56dfb10f66de22053e8789dd0e0d0d2473b16b86f3198b629c211d4a6"
Jul 17 05:33:13.013084 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:13.013062    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg_odh-ai-gateway-infra(f96bcef7-447e-4903-a512-8f7d7bdd5c64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64"
Jul 17 05:33:13.032930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:13.032904    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"]
Jul 17 05:33:14.190081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:14.190053    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:33:14.279430 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:14.279333    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96bcef7-447e-4903-a512-8f7d7bdd5c64-maas-api-service-ca\") pod \"f96bcef7-447e-4903-a512-8f7d7bdd5c64\" (UID: \"f96bcef7-447e-4903-a512-8f7d7bdd5c64\") "
Jul 17 05:33:14.279765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:14.279733    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96bcef7-447e-4903-a512-8f7d7bdd5c64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f96bcef7-447e-4903-a512-8f7d7bdd5c64" (UID: "f96bcef7-447e-4903-a512-8f7d7bdd5c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:33:14.380316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:14.380282    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f96bcef7-447e-4903-a512-8f7d7bdd5c64-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:33:15.024176 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:15.024096    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"
Jul 17 05:33:15.024176 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:15.024142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg" event={"ID":"f96bcef7-447e-4903-a512-8f7d7bdd5c64","Type":"ContainerDied","Data":"29877d29867cb5d0d8f9955a3b5f1bdbb8bac583e07d91cedac4b9b215168352"}
Jul 17 05:33:15.024414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:15.024201    2582 scope.go:117] "RemoveContainer" containerID="c52836e56dfb10f66de22053e8789dd0e0d0d2473b16b86f3198b629c211d4a6"
Jul 17 05:33:15.040160 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:15.040133    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"]
Jul 17 05:33:15.044369 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:15.044348    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-ntjsg"]
Jul 17 05:33:16.731172 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:16.731141    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" path="/var/lib/kubelet/pods/f96bcef7-447e-4903-a512-8f7d7bdd5c64/volumes"
Jul 17 05:33:27.101454 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101414    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"]
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101807    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101824    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101841    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101850    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101858    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101865    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101873    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101880    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101889    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101896    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101909    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101916    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101926    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.101934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.101936    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102015    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102026    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102039    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102046    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102055    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102063    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f811138a-14cb-4c16-a12a-c4c2ef399dc7" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102070    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:27.102641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.102082    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:27.106533 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.106510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:27.110600 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.110404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:33:27.110600 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.110402    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:33:27.111428 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.111401    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"]
Jul 17 05:33:27.196473 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.196424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7\" (UID: \"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:27.297794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.297755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7\" (UID: \"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:27.298576 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.298555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7\" (UID: \"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:27.418580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.418474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:27.549324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:27.549179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"]
Jul 17 05:33:27.551878 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:33:27.551841    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f8bea7_027a_4e1e_b4ea_a9ec25e06164.slice/crio-c4e5ec38c4ceabdbbc9f396a028000146f45346260eab64dee76e2c0de64b971 WatchSource:0}: Error finding container c4e5ec38c4ceabdbbc9f396a028000146f45346260eab64dee76e2c0de64b971: Status 404 returned error can't find the container with id c4e5ec38c4ceabdbbc9f396a028000146f45346260eab64dee76e2c0de64b971
Jul 17 05:33:28.087043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:28.086945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerStarted","Data":"c07115a001ba7c972d41f9fbfaa07a6819fbe488801f4cd2b6947a7490713335"}
Jul 17 05:33:28.087043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:28.087022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerStarted","Data":"c4e5ec38c4ceabdbbc9f396a028000146f45346260eab64dee76e2c0de64b971"}
Jul 17 05:33:28.102426 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:28.102374    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" podStartSLOduration=1.102358644 podStartE2EDuration="1.102358644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:33:28.10129809 +0000 UTC m=+2611.838360444" watchObservedRunningTime="2026-07-17 05:33:28.102358644 +0000 UTC m=+2611.839420979"
Jul 17 05:33:29.092690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:29.092655    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerID="c07115a001ba7c972d41f9fbfaa07a6819fbe488801f4cd2b6947a7490713335" exitCode=6
Jul 17 05:33:29.092876 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:29.092747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerDied","Data":"c07115a001ba7c972d41f9fbfaa07a6819fbe488801f4cd2b6947a7490713335"}
Jul 17 05:33:29.092940 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:29.092926    2582 scope.go:117] "RemoveContainer" containerID="c07115a001ba7c972d41f9fbfaa07a6819fbe488801f4cd2b6947a7490713335"
Jul 17 05:33:30.099058 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:30.099025    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerID="287e1caae423816fedba4fe87ac954705d46c9daea88a245e0a8029a0c148188" exitCode=6
Jul 17 05:33:30.099558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:30.099097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerDied","Data":"287e1caae423816fedba4fe87ac954705d46c9daea88a245e0a8029a0c148188"}
Jul 17 05:33:30.099558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:30.099153    2582 scope.go:117] "RemoveContainer" containerID="c07115a001ba7c972d41f9fbfaa07a6819fbe488801f4cd2b6947a7490713335"
Jul 17 05:33:30.099558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:30.099370    2582 scope.go:117] "RemoveContainer" containerID="287e1caae423816fedba4fe87ac954705d46c9daea88a245e0a8029a0c148188"
Jul 17 05:33:30.099737 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:30.099629    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7_odh-ai-gateway-infra(a7f8bea7-027a-4e1e-b4ea-a9ec25e06164)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164"
Jul 17 05:33:31.104688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:31.104662    2582 scope.go:117] "RemoveContainer" containerID="287e1caae423816fedba4fe87ac954705d46c9daea88a245e0a8029a0c148188"
Jul 17 05:33:31.105060 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:31.104826    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7_odh-ai-gateway-infra(a7f8bea7-027a-4e1e-b4ea-a9ec25e06164)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164"
Jul 17 05:33:39.048713 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.048676    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"]
Jul 17 05:33:39.049196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.048967    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:39.049196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.048976    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6251c7b-4922-452f-bb2d-375a26517277" containerName="revoke-keys"
Jul 17 05:33:39.049196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.048986    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:39.049196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.048992    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:39.049196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.049041    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96bcef7-447e-4903-a512-8f7d7bdd5c64" containerName="revoke-keys"
Jul 17 05:33:39.051973 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.051954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:39.059072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.059008    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"]
Jul 17 05:33:39.199089 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.199048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea309fbf-ae8b-41a8-a02e-284fd557831b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz\" (UID: \"ea309fbf-ae8b-41a8-a02e-284fd557831b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:39.300651 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.300557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea309fbf-ae8b-41a8-a02e-284fd557831b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz\" (UID: \"ea309fbf-ae8b-41a8-a02e-284fd557831b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:39.301198 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.301178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea309fbf-ae8b-41a8-a02e-284fd557831b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz\" (UID: \"ea309fbf-ae8b-41a8-a02e-284fd557831b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:39.363717 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.363686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:39.496094 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:39.496022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"]
Jul 17 05:33:39.498844 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:33:39.498816    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea309fbf_ae8b_41a8_a02e_284fd557831b.slice/crio-80e18f5bb194d524f56524f5522f918156fc32e9dc931cea1d95fcf038878663 WatchSource:0}: Error finding container 80e18f5bb194d524f56524f5522f918156fc32e9dc931cea1d95fcf038878663: Status 404 returned error can't find the container with id 80e18f5bb194d524f56524f5522f918156fc32e9dc931cea1d95fcf038878663
Jul 17 05:33:40.143959 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:40.143935    2582 generic.go:356] "Generic (PLEG): container finished" podID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerID="ce33d74cc5683cdb540261ee3879cb0bba6a9e7458a68a5d64737c3d90df968c" exitCode=6
Jul 17 05:33:40.144348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:40.143971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" event={"ID":"ea309fbf-ae8b-41a8-a02e-284fd557831b","Type":"ContainerDied","Data":"ce33d74cc5683cdb540261ee3879cb0bba6a9e7458a68a5d64737c3d90df968c"}
Jul 17 05:33:40.144348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:40.144006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" event={"ID":"ea309fbf-ae8b-41a8-a02e-284fd557831b","Type":"ContainerStarted","Data":"80e18f5bb194d524f56524f5522f918156fc32e9dc931cea1d95fcf038878663"}
Jul 17 05:33:40.144348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:40.144138    2582 scope.go:117] "RemoveContainer" containerID="ce33d74cc5683cdb540261ee3879cb0bba6a9e7458a68a5d64737c3d90df968c"
Jul 17 05:33:41.150411 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:41.150379    2582 generic.go:356] "Generic (PLEG): container finished" podID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerID="c7d642720420d37a6b496fadca8b2b3681f2c409077d22d1064e8f4f885f259c" exitCode=6
Jul 17 05:33:41.151144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:41.150475    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" event={"ID":"ea309fbf-ae8b-41a8-a02e-284fd557831b","Type":"ContainerDied","Data":"c7d642720420d37a6b496fadca8b2b3681f2c409077d22d1064e8f4f885f259c"}
Jul 17 05:33:41.151144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:41.150543    2582 scope.go:117] "RemoveContainer" containerID="ce33d74cc5683cdb540261ee3879cb0bba6a9e7458a68a5d64737c3d90df968c"
Jul 17 05:33:41.151144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:41.150638    2582 scope.go:117] "RemoveContainer" containerID="c7d642720420d37a6b496fadca8b2b3681f2c409077d22d1064e8f4f885f259c"
Jul 17 05:33:41.151144 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:41.150820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz_odh-ai-gateway-infra(ea309fbf-ae8b-41a8-a02e-284fd557831b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b"
Jul 17 05:33:42.142850 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.142815    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"]
Jul 17 05:33:42.147148 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.147131    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.150053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.150032    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e73f00c0\"/\"openshift-service-ca.crt\""
Jul 17 05:33:42.151341 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.151315    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e73f00c0\"/\"default-dockercfg-sjsht\""
Jul 17 05:33:42.151341 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.151332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e73f00c0\"/\"e2e-lifecycle-model-e73f00c0-kserve-self-signed-certs\""
Jul 17 05:33:42.151809 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.151318    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e73f00c0\"/\"kube-root-ca.crt\""
Jul 17 05:33:42.157861 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.157839    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"]
Jul 17 05:33:42.158168 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.158155    2582 scope.go:117] "RemoveContainer" containerID="c7d642720420d37a6b496fadca8b2b3681f2c409077d22d1064e8f4f885f259c"
Jul 17 05:33:42.158353 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:42.158333    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz_odh-ai-gateway-infra(ea309fbf-ae8b-41a8-a02e-284fd557831b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b"
Jul 17 05:33:42.230178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.230150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-home\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.230375 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.230184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-dshm\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.230375 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.230206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqm5f\" (UniqueName: \"kubernetes.io/projected/60358800-5f17-4568-b34b-7656665ec6a2-kube-api-access-fqm5f\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.230375 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.230240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-model-cache\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.230375 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.230281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-tmp-dir\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.230375 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.230340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60358800-5f17-4568-b34b-7656665ec6a2-tls-certs\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-home\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-dshm\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331647 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqm5f\" (UniqueName: \"kubernetes.io/projected/60358800-5f17-4568-b34b-7656665ec6a2-kube-api-access-fqm5f\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331647 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-model-cache\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331647 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-tmp-dir\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331647 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60358800-5f17-4568-b34b-7656665ec6a2-tls-certs\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-model-cache\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.331911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.331899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-home\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.332092 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.332077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-tmp-dir\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.333896 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.333871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-dshm\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.334472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.334445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60358800-5f17-4568-b34b-7656665ec6a2-tls-certs\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.341330 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.341250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqm5f\" (UniqueName: \"kubernetes.io/projected/60358800-5f17-4568-b34b-7656665ec6a2-kube-api-access-fqm5f\") pod \"e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") " pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.457146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.457104    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:42.587153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.587127    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"]
Jul 17 05:33:42.589622 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:33:42.589594    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60358800_5f17_4568_b34b_7656665ec6a2.slice/crio-9c5443d3724881df13559d1ffac71bd322cc7c7221409918a06268cdd062083e WatchSource:0}: Error finding container 9c5443d3724881df13559d1ffac71bd322cc7c7221409918a06268cdd062083e: Status 404 returned error can't find the container with id 9c5443d3724881df13559d1ffac71bd322cc7c7221409918a06268cdd062083e
Jul 17 05:33:42.730809 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:42.730740    2582 scope.go:117] "RemoveContainer" containerID="287e1caae423816fedba4fe87ac954705d46c9daea88a245e0a8029a0c148188"
Jul 17 05:33:43.165212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:43.165113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerStarted","Data":"ad798eaa4f32dd896c90551d64dddf915a9ad39476ce8f9cb6728c2f486a964b"}
Jul 17 05:33:43.166631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:43.166601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4" event={"ID":"60358800-5f17-4568-b34b-7656665ec6a2","Type":"ContainerStarted","Data":"8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72"}
Jul 17 05:33:43.166631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:43.166634    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4" event={"ID":"60358800-5f17-4568-b34b-7656665ec6a2","Type":"ContainerStarted","Data":"9c5443d3724881df13559d1ffac71bd322cc7c7221409918a06268cdd062083e"}
Jul 17 05:33:43.202160 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:43.202113    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4" podStartSLOduration=1.202098512 podStartE2EDuration="1.202098512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:33:43.200765301 +0000 UTC m=+2626.937827636" watchObservedRunningTime="2026-07-17 05:33:43.202098512 +0000 UTC m=+2626.939160847"
Jul 17 05:33:44.172471 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:44.172389    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerID="ad798eaa4f32dd896c90551d64dddf915a9ad39476ce8f9cb6728c2f486a964b" exitCode=6
Jul 17 05:33:44.172911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:44.172465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerDied","Data":"ad798eaa4f32dd896c90551d64dddf915a9ad39476ce8f9cb6728c2f486a964b"}
Jul 17 05:33:44.172911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:44.172527    2582 scope.go:117] "RemoveContainer" containerID="287e1caae423816fedba4fe87ac954705d46c9daea88a245e0a8029a0c148188"
Jul 17 05:33:44.172911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:44.172831    2582 scope.go:117] "RemoveContainer" containerID="ad798eaa4f32dd896c90551d64dddf915a9ad39476ce8f9cb6728c2f486a964b"
Jul 17 05:33:44.173068 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:44.173010    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7_odh-ai-gateway-infra(a7f8bea7-027a-4e1e-b4ea-a9ec25e06164)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164"
Jul 17 05:33:44.196835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:44.196807    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"]
Jul 17 05:33:45.307828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:45.307802    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:45.460212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:45.460179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164-maas-api-service-ca\") pod \"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164\" (UID: \"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164\") "
Jul 17 05:33:45.460568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:45.460545    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" (UID: "a7f8bea7-027a-4e1e-b4ea-a9ec25e06164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:33:45.560734 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:45.560701    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:33:46.183145 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:46.183112    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"
Jul 17 05:33:46.183145 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:46.183122    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7" event={"ID":"a7f8bea7-027a-4e1e-b4ea-a9ec25e06164","Type":"ContainerDied","Data":"c4e5ec38c4ceabdbbc9f396a028000146f45346260eab64dee76e2c0de64b971"}
Jul 17 05:33:46.183376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:46.183186    2582 scope.go:117] "RemoveContainer" containerID="ad798eaa4f32dd896c90551d64dddf915a9ad39476ce8f9cb6728c2f486a964b"
Jul 17 05:33:46.204852 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:46.204823    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"]
Jul 17 05:33:46.207979 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:46.207956    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-znwt7"]
Jul 17 05:33:46.730396 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:46.730364    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" path="/var/lib/kubelet/pods/a7f8bea7-027a-4e1e-b4ea-a9ec25e06164/volumes"
Jul 17 05:33:52.457964 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:52.457932    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:52.458413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:52.458134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:52.470792 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:52.470768    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:53.225540 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:53.225512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:33:54.489122 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489081    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"]
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489409    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489421    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489435    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489441    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489448    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489453    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489511    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.489527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.489518    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:33:54.493856 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.493839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:33:54.498752 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.498725    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"]
Jul 17 05:33:54.534074 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.534037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5f6e-afa1-45b8-80bf-a5b492f8e481-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9\" (UID: \"484e5f6e-afa1-45b8-80bf-a5b492f8e481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:33:54.634707 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.634671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5f6e-afa1-45b8-80bf-a5b492f8e481-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9\" (UID: \"484e5f6e-afa1-45b8-80bf-a5b492f8e481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:33:54.635316 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.635294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5f6e-afa1-45b8-80bf-a5b492f8e481-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9\" (UID: \"484e5f6e-afa1-45b8-80bf-a5b492f8e481\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:33:54.805625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.805536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:33:54.936984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:54.936964    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"]
Jul 17 05:33:54.939168 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:33:54.939142    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484e5f6e_afa1_45b8_80bf_a5b492f8e481.slice/crio-dc84a119f8c2cce57657dafe5067d5aefb57ac84d1fcd745feebfad083c2c698 WatchSource:0}: Error finding container dc84a119f8c2cce57657dafe5067d5aefb57ac84d1fcd745feebfad083c2c698: Status 404 returned error can't find the container with id dc84a119f8c2cce57657dafe5067d5aefb57ac84d1fcd745feebfad083c2c698
Jul 17 05:33:55.223623 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:55.223584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerStarted","Data":"493a0c9a7b9afac3e1788933c7a84fe617d93d98cca5345b20edb48bcc271a1d"}
Jul 17 05:33:55.223804 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:55.223636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerStarted","Data":"dc84a119f8c2cce57657dafe5067d5aefb57ac84d1fcd745feebfad083c2c698"}
Jul 17 05:33:55.725241 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:55.725211    2582 scope.go:117] "RemoveContainer" containerID="c7d642720420d37a6b496fadca8b2b3681f2c409077d22d1064e8f4f885f259c"
Jul 17 05:33:56.229698 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:56.229654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" event={"ID":"ea309fbf-ae8b-41a8-a02e-284fd557831b","Type":"ContainerStarted","Data":"167edfb292f0d44bad114db3f3c663a62b5690e5df739b484573365fff1a2649"}
Jul 17 05:33:56.231080 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:56.231058    2582 generic.go:356] "Generic (PLEG): container finished" podID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerID="493a0c9a7b9afac3e1788933c7a84fe617d93d98cca5345b20edb48bcc271a1d" exitCode=6
Jul 17 05:33:56.231217 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:56.231139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerDied","Data":"493a0c9a7b9afac3e1788933c7a84fe617d93d98cca5345b20edb48bcc271a1d"}
Jul 17 05:33:56.231304 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:56.231291    2582 scope.go:117] "RemoveContainer" containerID="493a0c9a7b9afac3e1788933c7a84fe617d93d98cca5345b20edb48bcc271a1d"
Jul 17 05:33:56.245059 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:56.244899    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" podStartSLOduration=17.244881884 podStartE2EDuration="17.244881884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:33:56.24485741 +0000 UTC m=+2639.981919747" watchObservedRunningTime="2026-07-17 05:33:56.244881884 +0000 UTC m=+2639.981944222"
Jul 17 05:33:56.245676 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:56.245640    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" podStartSLOduration=2.245627447 podStartE2EDuration="2.245627447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:33:55.246020547 +0000 UTC m=+2638.983082882" watchObservedRunningTime="2026-07-17 05:33:56.245627447 +0000 UTC m=+2639.982689783"
Jul 17 05:33:57.242854 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.242824    2582 generic.go:356] "Generic (PLEG): container finished" podID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerID="167edfb292f0d44bad114db3f3c663a62b5690e5df739b484573365fff1a2649" exitCode=6
Jul 17 05:33:57.243301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.242900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" event={"ID":"ea309fbf-ae8b-41a8-a02e-284fd557831b","Type":"ContainerDied","Data":"167edfb292f0d44bad114db3f3c663a62b5690e5df739b484573365fff1a2649"}
Jul 17 05:33:57.243301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.242987    2582 scope.go:117] "RemoveContainer" containerID="c7d642720420d37a6b496fadca8b2b3681f2c409077d22d1064e8f4f885f259c"
Jul 17 05:33:57.243301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.243297    2582 scope.go:117] "RemoveContainer" containerID="167edfb292f0d44bad114db3f3c663a62b5690e5df739b484573365fff1a2649"
Jul 17 05:33:57.243538 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:57.243510    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz_odh-ai-gateway-infra(ea309fbf-ae8b-41a8-a02e-284fd557831b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b"
Jul 17 05:33:57.244818 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.244796    2582 generic.go:356] "Generic (PLEG): container finished" podID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerID="2ca14078e7ce9e217b9636f820d5059231ebf82412399b884abc33b094b6edd0" exitCode=6
Jul 17 05:33:57.244928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.244879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerDied","Data":"2ca14078e7ce9e217b9636f820d5059231ebf82412399b884abc33b094b6edd0"}
Jul 17 05:33:57.245066 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.245054    2582 scope.go:117] "RemoveContainer" containerID="2ca14078e7ce9e217b9636f820d5059231ebf82412399b884abc33b094b6edd0"
Jul 17 05:33:57.245221 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:57.245206    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9_odh-ai-gateway-infra(484e5f6e-afa1-45b8-80bf-a5b492f8e481)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481"
Jul 17 05:33:57.254972 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.254949    2582 scope.go:117] "RemoveContainer" containerID="493a0c9a7b9afac3e1788933c7a84fe617d93d98cca5345b20edb48bcc271a1d"
Jul 17 05:33:57.257040 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:57.257021    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"]
Jul 17 05:33:58.252577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:58.252553    2582 scope.go:117] "RemoveContainer" containerID="2ca14078e7ce9e217b9636f820d5059231ebf82412399b884abc33b094b6edd0"
Jul 17 05:33:58.252929 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:33:58.252808    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9_odh-ai-gateway-infra(484e5f6e-afa1-45b8-80bf-a5b492f8e481)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481"
Jul 17 05:33:58.381254 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:58.381227    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:58.465754 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:58.465718    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea309fbf-ae8b-41a8-a02e-284fd557831b-maas-api-service-ca\") pod \"ea309fbf-ae8b-41a8-a02e-284fd557831b\" (UID: \"ea309fbf-ae8b-41a8-a02e-284fd557831b\") "
Jul 17 05:33:58.466086 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:58.466059    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea309fbf-ae8b-41a8-a02e-284fd557831b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea309fbf-ae8b-41a8-a02e-284fd557831b" (UID: "ea309fbf-ae8b-41a8-a02e-284fd557831b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:33:58.567001 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:58.566916    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea309fbf-ae8b-41a8-a02e-284fd557831b-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:33:59.257979 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:59.257953    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"
Jul 17 05:33:59.257979 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:59.257970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz" event={"ID":"ea309fbf-ae8b-41a8-a02e-284fd557831b","Type":"ContainerDied","Data":"80e18f5bb194d524f56524f5522f918156fc32e9dc931cea1d95fcf038878663"}
Jul 17 05:33:59.258586 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:59.258028    2582 scope.go:117] "RemoveContainer" containerID="167edfb292f0d44bad114db3f3c663a62b5690e5df739b484573365fff1a2649"
Jul 17 05:33:59.275461 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:59.275433    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"]
Jul 17 05:33:59.279026 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:33:59.279000    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-sbngz"]
Jul 17 05:34:00.730587 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:00.730556    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" path="/var/lib/kubelet/pods/ea309fbf-ae8b-41a8-a02e-284fd557831b/volumes"
Jul 17 05:34:08.724912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:08.724878    2582 scope.go:117] "RemoveContainer" containerID="2ca14078e7ce9e217b9636f820d5059231ebf82412399b884abc33b094b6edd0"
Jul 17 05:34:09.303470 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:09.303385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerStarted","Data":"b15110dc9e44d6f0eb63d28d4000d51d9e36b724238bb136c93f381410bd3723"}
Jul 17 05:34:10.309621 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:10.309590    2582 generic.go:356] "Generic (PLEG): container finished" podID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerID="b15110dc9e44d6f0eb63d28d4000d51d9e36b724238bb136c93f381410bd3723" exitCode=6
Jul 17 05:34:10.310393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:10.309662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerDied","Data":"b15110dc9e44d6f0eb63d28d4000d51d9e36b724238bb136c93f381410bd3723"}
Jul 17 05:34:10.310393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:10.309723    2582 scope.go:117] "RemoveContainer" containerID="2ca14078e7ce9e217b9636f820d5059231ebf82412399b884abc33b094b6edd0"
Jul 17 05:34:10.310393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:10.310016    2582 scope.go:117] "RemoveContainer" containerID="b15110dc9e44d6f0eb63d28d4000d51d9e36b724238bb136c93f381410bd3723"
Jul 17 05:34:10.310393 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:10.310287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9_odh-ai-gateway-infra(484e5f6e-afa1-45b8-80bf-a5b492f8e481)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481"
Jul 17 05:34:10.338080 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:10.338051    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"]
Jul 17 05:34:11.447429 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:11.447396    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:34:11.469954 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:11.469920    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5f6e-afa1-45b8-80bf-a5b492f8e481-maas-api-service-ca\") pod \"484e5f6e-afa1-45b8-80bf-a5b492f8e481\" (UID: \"484e5f6e-afa1-45b8-80bf-a5b492f8e481\") "
Jul 17 05:34:11.470282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:11.470242    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484e5f6e-afa1-45b8-80bf-a5b492f8e481-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "484e5f6e-afa1-45b8-80bf-a5b492f8e481" (UID: "484e5f6e-afa1-45b8-80bf-a5b492f8e481"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:34:11.570854 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:11.570769    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/484e5f6e-afa1-45b8-80bf-a5b492f8e481-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:12.321037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:12.320998    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9" event={"ID":"484e5f6e-afa1-45b8-80bf-a5b492f8e481","Type":"ContainerDied","Data":"dc84a119f8c2cce57657dafe5067d5aefb57ac84d1fcd745feebfad083c2c698"}
Jul 17 05:34:12.321037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:12.321030    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"
Jul 17 05:34:12.321310 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:12.321061    2582 scope.go:117] "RemoveContainer" containerID="b15110dc9e44d6f0eb63d28d4000d51d9e36b724238bb136c93f381410bd3723"
Jul 17 05:34:12.345083 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:12.345057    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"]
Jul 17 05:34:12.348235 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:12.348210    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-bv7l9"]
Jul 17 05:34:12.730639 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:12.730600    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" path="/var/lib/kubelet/pods/484e5f6e-afa1-45b8-80bf-a5b492f8e481/volumes"
Jul 17 05:34:17.146035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.145998    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"]
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146292    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146305    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146319    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146326    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146340    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146349    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146357    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146433    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f8bea7-027a-4e1e-b4ea-a9ec25e06164" containerName="revoke-keys"
Jul 17 05:34:17.146730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146443    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:17.146730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146450    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146456    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146462    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:17.146730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.146471    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:17.150853 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.150836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:17.153569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.153551    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:34:17.153569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.153559    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:34:17.156730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.156704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"]
Jul 17 05:34:17.215832 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.215799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa70cf7-bd82-4020-8291-2cb59eaeefa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f\" (UID: \"bfa70cf7-bd82-4020-8291-2cb59eaeefa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:17.316883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.316842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa70cf7-bd82-4020-8291-2cb59eaeefa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f\" (UID: \"bfa70cf7-bd82-4020-8291-2cb59eaeefa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:17.317575 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.317553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa70cf7-bd82-4020-8291-2cb59eaeefa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f\" (UID: \"bfa70cf7-bd82-4020-8291-2cb59eaeefa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:17.462053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.462023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:17.592100 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:17.592072    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"]
Jul 17 05:34:17.594383 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:34:17.594357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa70cf7_bd82_4020_8291_2cb59eaeefa9.slice/crio-2fa7711afd6d242977b7d23f701b749411b2db3fa6fe23228c4c9417d1cc6715 WatchSource:0}: Error finding container 2fa7711afd6d242977b7d23f701b749411b2db3fa6fe23228c4c9417d1cc6715: Status 404 returned error can't find the container with id 2fa7711afd6d242977b7d23f701b749411b2db3fa6fe23228c4c9417d1cc6715
Jul 17 05:34:18.349788 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:18.349758    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerID="5f46f93320e0c3b176713d2b89580c1adbe99a3b2fe5bd65bfc56ecc45db63ad" exitCode=6
Jul 17 05:34:18.350194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:18.349801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" event={"ID":"bfa70cf7-bd82-4020-8291-2cb59eaeefa9","Type":"ContainerDied","Data":"5f46f93320e0c3b176713d2b89580c1adbe99a3b2fe5bd65bfc56ecc45db63ad"}
Jul 17 05:34:18.350194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:18.349860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" event={"ID":"bfa70cf7-bd82-4020-8291-2cb59eaeefa9","Type":"ContainerStarted","Data":"2fa7711afd6d242977b7d23f701b749411b2db3fa6fe23228c4c9417d1cc6715"}
Jul 17 05:34:18.350194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:18.350058    2582 scope.go:117] "RemoveContainer" containerID="5f46f93320e0c3b176713d2b89580c1adbe99a3b2fe5bd65bfc56ecc45db63ad"
Jul 17 05:34:19.355701 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:19.355615    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerID="34314f35b1cccc1e60963c1118e4e5c848530866919306a4c3636713c5ed7423" exitCode=6
Jul 17 05:34:19.355701 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:19.355677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" event={"ID":"bfa70cf7-bd82-4020-8291-2cb59eaeefa9","Type":"ContainerDied","Data":"34314f35b1cccc1e60963c1118e4e5c848530866919306a4c3636713c5ed7423"}
Jul 17 05:34:19.356122 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:19.355735    2582 scope.go:117] "RemoveContainer" containerID="5f46f93320e0c3b176713d2b89580c1adbe99a3b2fe5bd65bfc56ecc45db63ad"
Jul 17 05:34:19.356122 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:19.355914    2582 scope.go:117] "RemoveContainer" containerID="34314f35b1cccc1e60963c1118e4e5c848530866919306a4c3636713c5ed7423"
Jul 17 05:34:19.356199 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:19.356120    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f_odh-ai-gateway-infra(bfa70cf7-bd82-4020-8291-2cb59eaeefa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9"
Jul 17 05:34:20.361330 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:20.361298    2582 scope.go:117] "RemoveContainer" containerID="34314f35b1cccc1e60963c1118e4e5c848530866919306a4c3636713c5ed7423"
Jul 17 05:34:20.361712 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:20.361472    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f_odh-ai-gateway-infra(bfa70cf7-bd82-4020-8291-2cb59eaeefa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9"
Jul 17 05:34:27.681144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:27.681106    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"]
Jul 17 05:34:35.725142 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:35.725113    2582 scope.go:117] "RemoveContainer" containerID="34314f35b1cccc1e60963c1118e4e5c848530866919306a4c3636713c5ed7423"
Jul 17 05:34:36.432172 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:36.432134    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerID="607237ad0ad1d3d261869913b9069cc3127f6e991be07b03b41e7354af2c0ef9" exitCode=6
Jul 17 05:34:36.432393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:36.432212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" event={"ID":"bfa70cf7-bd82-4020-8291-2cb59eaeefa9","Type":"ContainerDied","Data":"607237ad0ad1d3d261869913b9069cc3127f6e991be07b03b41e7354af2c0ef9"}
Jul 17 05:34:36.432393 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:36.432342    2582 scope.go:117] "RemoveContainer" containerID="34314f35b1cccc1e60963c1118e4e5c848530866919306a4c3636713c5ed7423"
Jul 17 05:34:36.432683 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:36.432662    2582 scope.go:117] "RemoveContainer" containerID="607237ad0ad1d3d261869913b9069cc3127f6e991be07b03b41e7354af2c0ef9"
Jul 17 05:34:36.432945 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:36.432915    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f_odh-ai-gateway-infra(bfa70cf7-bd82-4020-8291-2cb59eaeefa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9"
Jul 17 05:34:37.457821 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:37.457789    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"]
Jul 17 05:34:37.588988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:37.588960    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:37.690492 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:37.690455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa70cf7-bd82-4020-8291-2cb59eaeefa9-maas-api-service-ca\") pod \"bfa70cf7-bd82-4020-8291-2cb59eaeefa9\" (UID: \"bfa70cf7-bd82-4020-8291-2cb59eaeefa9\") "
Jul 17 05:34:37.690840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:37.690816    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa70cf7-bd82-4020-8291-2cb59eaeefa9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfa70cf7-bd82-4020-8291-2cb59eaeefa9" (UID: "bfa70cf7-bd82-4020-8291-2cb59eaeefa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:34:37.791630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:37.791545    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa70cf7-bd82-4020-8291-2cb59eaeefa9-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:38.444472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:38.444440    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"
Jul 17 05:34:38.444472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:38.444459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f" event={"ID":"bfa70cf7-bd82-4020-8291-2cb59eaeefa9","Type":"ContainerDied","Data":"2fa7711afd6d242977b7d23f701b749411b2db3fa6fe23228c4c9417d1cc6715"}
Jul 17 05:34:38.444705 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:38.444517    2582 scope.go:117] "RemoveContainer" containerID="607237ad0ad1d3d261869913b9069cc3127f6e991be07b03b41e7354af2c0ef9"
Jul 17 05:34:38.465043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:38.465018    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"]
Jul 17 05:34:38.470786 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:38.470764    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-tcx7f"]
Jul 17 05:34:38.729804 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:38.729725    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" path="/var/lib/kubelet/pods/bfa70cf7-bd82-4020-8291-2cb59eaeefa9/volumes"
Jul 17 05:34:42.701492 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:42.701450    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4" podUID="60358800-5f17-4568-b34b-7656665ec6a2" containerName="main" containerID="cri-o://8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72" gracePeriod=45
Jul 17 05:34:42.948712 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:42.948680    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:34:43.032054 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.031956    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-home\") pod \"60358800-5f17-4568-b34b-7656665ec6a2\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") "
Jul 17 05:34:43.032054 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-dshm\") pod \"60358800-5f17-4568-b34b-7656665ec6a2\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") "
Jul 17 05:34:43.032054 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-model-cache\") pod \"60358800-5f17-4568-b34b-7656665ec6a2\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") "
Jul 17 05:34:43.032445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032075    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-tmp-dir\") pod \"60358800-5f17-4568-b34b-7656665ec6a2\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") "
Jul 17 05:34:43.032445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqm5f\" (UniqueName: \"kubernetes.io/projected/60358800-5f17-4568-b34b-7656665ec6a2-kube-api-access-fqm5f\") pod \"60358800-5f17-4568-b34b-7656665ec6a2\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") "
Jul 17 05:34:43.032445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60358800-5f17-4568-b34b-7656665ec6a2-tls-certs\") pod \"60358800-5f17-4568-b34b-7656665ec6a2\" (UID: \"60358800-5f17-4568-b34b-7656665ec6a2\") "
Jul 17 05:34:43.032445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032306    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-home" (OuterVolumeSpecName: "home") pod "60358800-5f17-4568-b34b-7656665ec6a2" (UID: "60358800-5f17-4568-b34b-7656665ec6a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:34:43.032445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "60358800-5f17-4568-b34b-7656665ec6a2" (UID: "60358800-5f17-4568-b34b-7656665ec6a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:34:43.032445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.032383    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60358800-5f17-4568-b34b-7656665ec6a2" (UID: "60358800-5f17-4568-b34b-7656665ec6a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:34:43.034560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.034534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60358800-5f17-4568-b34b-7656665ec6a2-kube-api-access-fqm5f" (OuterVolumeSpecName: "kube-api-access-fqm5f") pod "60358800-5f17-4568-b34b-7656665ec6a2" (UID: "60358800-5f17-4568-b34b-7656665ec6a2"). InnerVolumeSpecName "kube-api-access-fqm5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:34:43.034663 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.034554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60358800-5f17-4568-b34b-7656665ec6a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60358800-5f17-4568-b34b-7656665ec6a2" (UID: "60358800-5f17-4568-b34b-7656665ec6a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:34:43.034663 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.034574    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-dshm" (OuterVolumeSpecName: "dshm") pod "60358800-5f17-4568-b34b-7656665ec6a2" (UID: "60358800-5f17-4568-b34b-7656665ec6a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:34:43.133043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.133004    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-dshm\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:43.133043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.133036    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-model-cache\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:43.133252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.133067    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-tmp-dir\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:43.133252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.133077    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqm5f\" (UniqueName: \"kubernetes.io/projected/60358800-5f17-4568-b34b-7656665ec6a2-kube-api-access-fqm5f\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:43.133252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.133086    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60358800-5f17-4568-b34b-7656665ec6a2-tls-certs\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:43.133252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.133094    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60358800-5f17-4568-b34b-7656665ec6a2-home\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:34:43.467317 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.467276    2582 generic.go:356] "Generic (PLEG): container finished" podID="60358800-5f17-4568-b34b-7656665ec6a2" containerID="8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72" exitCode=0
Jul 17 05:34:43.467525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.467337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"
Jul 17 05:34:43.467525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.467365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4" event={"ID":"60358800-5f17-4568-b34b-7656665ec6a2","Type":"ContainerDied","Data":"8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72"}
Jul 17 05:34:43.467525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.467411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4" event={"ID":"60358800-5f17-4568-b34b-7656665ec6a2","Type":"ContainerDied","Data":"9c5443d3724881df13559d1ffac71bd322cc7c7221409918a06268cdd062083e"}
Jul 17 05:34:43.467525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.467430    2582 scope.go:117] "RemoveContainer" containerID="8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72"
Jul 17 05:34:43.475794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.475778    2582 scope.go:117] "RemoveContainer" containerID="8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72"
Jul 17 05:34:43.476067 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:43.476051    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72\": container with ID starting with 8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72 not found: ID does not exist" containerID="8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72"
Jul 17 05:34:43.476128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.476075    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72"} err="failed to get container status \"8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72\": rpc error: code = NotFound desc = could not find container \"8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72\": container with ID starting with 8705cf412a3c8d52f1d4758381e7d3c2a68c24859c2162ee2e6b7b605b54cf72 not found: ID does not exist"
Jul 17 05:34:43.488205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.488182    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"]
Jul 17 05:34:43.493839 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:43.493817    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e73f00c0/e2e-lifecycle-model-e73f00c0-kserve-78c7585f9-n6xf4"]
Jul 17 05:34:44.533233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533193    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"]
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533508    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533522    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea309fbf-ae8b-41a8-a02e-284fd557831b" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533534    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533539    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533546    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533551    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533569    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533580    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60358800-5f17-4568-b34b-7656665ec6a2" containerName="main"
Jul 17 05:34:44.533630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533585    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60358800-5f17-4568-b34b-7656665ec6a2" containerName="main"
Jul 17 05:34:44.533947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533638    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60358800-5f17-4568-b34b-7656665ec6a2" containerName="main"
Jul 17 05:34:44.533947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533645    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e5f6e-afa1-45b8-80bf-a5b492f8e481" containerName="revoke-keys"
Jul 17 05:34:44.533947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533651    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.533947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533656    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.533947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.533662    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:34:44.537912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.537892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:34:44.540656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.540637    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:34:44.540656 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.540652    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:34:44.543791 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.543770    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"]
Jul 17 05:34:44.646225 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.646183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed2466-fd0b-4d23-8dee-2966f03db083-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc\" (UID: \"04ed2466-fd0b-4d23-8dee-2966f03db083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:34:44.730195 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.730164    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60358800-5f17-4568-b34b-7656665ec6a2" path="/var/lib/kubelet/pods/60358800-5f17-4568-b34b-7656665ec6a2/volumes"
Jul 17 05:34:44.747606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.747581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed2466-fd0b-4d23-8dee-2966f03db083-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc\" (UID: \"04ed2466-fd0b-4d23-8dee-2966f03db083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:34:44.748169 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.748149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed2466-fd0b-4d23-8dee-2966f03db083-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc\" (UID: \"04ed2466-fd0b-4d23-8dee-2966f03db083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:34:44.849288 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.849201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:34:44.985625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:44.985599    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"]
Jul 17 05:34:44.988018 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:34:44.987992    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ed2466_fd0b_4d23_8dee_2966f03db083.slice/crio-f9d2a124813f162eef52b3d81b51df63e2dc8166ad72da26ee88e5873b6b0fd9 WatchSource:0}: Error finding container f9d2a124813f162eef52b3d81b51df63e2dc8166ad72da26ee88e5873b6b0fd9: Status 404 returned error can't find the container with id f9d2a124813f162eef52b3d81b51df63e2dc8166ad72da26ee88e5873b6b0fd9
Jul 17 05:34:45.477457 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:45.477367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" event={"ID":"04ed2466-fd0b-4d23-8dee-2966f03db083","Type":"ContainerStarted","Data":"dc74a9387cc71c7e11e83b30ed22d1977be96b683e4a77fe7b7bb06e8f2caf36"}
Jul 17 05:34:45.477457 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:45.477421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" event={"ID":"04ed2466-fd0b-4d23-8dee-2966f03db083","Type":"ContainerStarted","Data":"f9d2a124813f162eef52b3d81b51df63e2dc8166ad72da26ee88e5873b6b0fd9"}
Jul 17 05:34:45.493584 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:45.493527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" podStartSLOduration=1.493506492 podStartE2EDuration="1.493506492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:34:45.493011119 +0000 UTC m=+2689.230073457" watchObservedRunningTime="2026-07-17 05:34:45.493506492 +0000 UTC m=+2689.230568829"
Jul 17 05:34:46.488899 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:46.488864    2582 generic.go:356] "Generic (PLEG): container finished" podID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerID="dc74a9387cc71c7e11e83b30ed22d1977be96b683e4a77fe7b7bb06e8f2caf36" exitCode=6
Jul 17 05:34:46.489418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:46.488910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" event={"ID":"04ed2466-fd0b-4d23-8dee-2966f03db083","Type":"ContainerDied","Data":"dc74a9387cc71c7e11e83b30ed22d1977be96b683e4a77fe7b7bb06e8f2caf36"}
Jul 17 05:34:46.489418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:46.489167    2582 scope.go:117] "RemoveContainer" containerID="dc74a9387cc71c7e11e83b30ed22d1977be96b683e4a77fe7b7bb06e8f2caf36"
Jul 17 05:34:47.494775 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:47.494738    2582 generic.go:356] "Generic (PLEG): container finished" podID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerID="d5dacecdc6962e1dc86a8caa0b79ec821ba7cd51e742efd33a4413118be55765" exitCode=6
Jul 17 05:34:47.495226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:47.494781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" event={"ID":"04ed2466-fd0b-4d23-8dee-2966f03db083","Type":"ContainerDied","Data":"d5dacecdc6962e1dc86a8caa0b79ec821ba7cd51e742efd33a4413118be55765"}
Jul 17 05:34:47.495226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:47.494841    2582 scope.go:117] "RemoveContainer" containerID="dc74a9387cc71c7e11e83b30ed22d1977be96b683e4a77fe7b7bb06e8f2caf36"
Jul 17 05:34:47.495226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:47.495012    2582 scope.go:117] "RemoveContainer" containerID="d5dacecdc6962e1dc86a8caa0b79ec821ba7cd51e742efd33a4413118be55765"
Jul 17 05:34:47.495377 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:47.495236    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc_odh-ai-gateway-infra(04ed2466-fd0b-4d23-8dee-2966f03db083)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083"
Jul 17 05:34:48.501343 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:48.501315    2582 scope.go:117] "RemoveContainer" containerID="d5dacecdc6962e1dc86a8caa0b79ec821ba7cd51e742efd33a4413118be55765"
Jul 17 05:34:48.501734 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:34:48.501504    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc_odh-ai-gateway-infra(04ed2466-fd0b-4d23-8dee-2966f03db083)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083"
Jul 17 05:34:56.837013 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:56.836980    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:34:56.840886 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:34:56.840868    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:35:00.996509 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:00.996478    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"]
Jul 17 05:35:00.996984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:00.996802    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:35:00.996984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:00.996814    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa70cf7-bd82-4020-8291-2cb59eaeefa9" containerName="revoke-keys"
Jul 17 05:35:01.001348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.001327    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:01.009296 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.009214    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"]
Jul 17 05:35:01.079485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.079447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009a53a-f9ec-4068-b394-41d25b11cd01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-r68jm\" (UID: \"a009a53a-f9ec-4068-b394-41d25b11cd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:01.180568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.180534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009a53a-f9ec-4068-b394-41d25b11cd01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-r68jm\" (UID: \"a009a53a-f9ec-4068-b394-41d25b11cd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:01.181123 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.181102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009a53a-f9ec-4068-b394-41d25b11cd01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-r68jm\" (UID: \"a009a53a-f9ec-4068-b394-41d25b11cd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:01.312101 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.312012    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:01.443137 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.443111    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"]
Jul 17 05:35:01.445527 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:35:01.445504    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda009a53a_f9ec_4068_b394_41d25b11cd01.slice/crio-9880a1857eb56ada2fa445255d773772f5f9ee23f14f79095d612d33289e8db4 WatchSource:0}: Error finding container 9880a1857eb56ada2fa445255d773772f5f9ee23f14f79095d612d33289e8db4: Status 404 returned error can't find the container with id 9880a1857eb56ada2fa445255d773772f5f9ee23f14f79095d612d33289e8db4
Jul 17 05:35:01.560370 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:01.560337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" event={"ID":"a009a53a-f9ec-4068-b394-41d25b11cd01","Type":"ContainerStarted","Data":"9880a1857eb56ada2fa445255d773772f5f9ee23f14f79095d612d33289e8db4"}
Jul 17 05:35:02.565998 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:02.565967    2582 generic.go:356] "Generic (PLEG): container finished" podID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerID="19c26909529226230592a5ab057c3259cf0296c17844874fcd7aaabbfe19cf27" exitCode=6
Jul 17 05:35:02.566462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:02.566042    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" event={"ID":"a009a53a-f9ec-4068-b394-41d25b11cd01","Type":"ContainerDied","Data":"19c26909529226230592a5ab057c3259cf0296c17844874fcd7aaabbfe19cf27"}
Jul 17 05:35:02.566462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:02.566215    2582 scope.go:117] "RemoveContainer" containerID="19c26909529226230592a5ab057c3259cf0296c17844874fcd7aaabbfe19cf27"
Jul 17 05:35:02.725485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:02.725454    2582 scope.go:117] "RemoveContainer" containerID="d5dacecdc6962e1dc86a8caa0b79ec821ba7cd51e742efd33a4413118be55765"
Jul 17 05:35:03.571624 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.571534    2582 generic.go:356] "Generic (PLEG): container finished" podID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerID="b701ad873cc692f5d9dafc0ae92d036a195422c213e261b0d10dc3d158c4be2c" exitCode=6
Jul 17 05:35:03.572072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.571624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" event={"ID":"a009a53a-f9ec-4068-b394-41d25b11cd01","Type":"ContainerDied","Data":"b701ad873cc692f5d9dafc0ae92d036a195422c213e261b0d10dc3d158c4be2c"}
Jul 17 05:35:03.572072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.571724    2582 scope.go:117] "RemoveContainer" containerID="19c26909529226230592a5ab057c3259cf0296c17844874fcd7aaabbfe19cf27"
Jul 17 05:35:03.572072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.571802    2582 scope.go:117] "RemoveContainer" containerID="b701ad873cc692f5d9dafc0ae92d036a195422c213e261b0d10dc3d158c4be2c"
Jul 17 05:35:03.572072 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:03.572043    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-r68jm_odh-ai-gateway-infra(a009a53a-f9ec-4068-b394-41d25b11cd01)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01"
Jul 17 05:35:03.573593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.573572    2582 generic.go:356] "Generic (PLEG): container finished" podID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerID="5a7b9f1d1ae37965974c5bd4fb706b0104f6f901f2ec5c0c7dfa6dabf47da659" exitCode=6
Jul 17 05:35:03.573751 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.573643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" event={"ID":"04ed2466-fd0b-4d23-8dee-2966f03db083","Type":"ContainerDied","Data":"5a7b9f1d1ae37965974c5bd4fb706b0104f6f901f2ec5c0c7dfa6dabf47da659"}
Jul 17 05:35:03.573918 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.573905    2582 scope.go:117] "RemoveContainer" containerID="5a7b9f1d1ae37965974c5bd4fb706b0104f6f901f2ec5c0c7dfa6dabf47da659"
Jul 17 05:35:03.574073 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:03.574056    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc_odh-ai-gateway-infra(04ed2466-fd0b-4d23-8dee-2966f03db083)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083"
Jul 17 05:35:03.584506 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:03.584490    2582 scope.go:117] "RemoveContainer" containerID="d5dacecdc6962e1dc86a8caa0b79ec821ba7cd51e742efd33a4413118be55765"
Jul 17 05:35:04.581361 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:04.581327    2582 scope.go:117] "RemoveContainer" containerID="b701ad873cc692f5d9dafc0ae92d036a195422c213e261b0d10dc3d158c4be2c"
Jul 17 05:35:04.581746 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:04.581545    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-r68jm_odh-ai-gateway-infra(a009a53a-f9ec-4068-b394-41d25b11cd01)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01"
Jul 17 05:35:04.614534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:04.614507    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"]
Jul 17 05:35:04.740917 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:04.740891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:35:04.810230 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:04.810201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed2466-fd0b-4d23-8dee-2966f03db083-maas-api-service-ca\") pod \"04ed2466-fd0b-4d23-8dee-2966f03db083\" (UID: \"04ed2466-fd0b-4d23-8dee-2966f03db083\") "
Jul 17 05:35:04.810601 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:04.810576    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed2466-fd0b-4d23-8dee-2966f03db083-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04ed2466-fd0b-4d23-8dee-2966f03db083" (UID: "04ed2466-fd0b-4d23-8dee-2966f03db083"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:35:04.911655 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:04.911574    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed2466-fd0b-4d23-8dee-2966f03db083-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:35:05.586906 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:05.586876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc" event={"ID":"04ed2466-fd0b-4d23-8dee-2966f03db083","Type":"ContainerDied","Data":"f9d2a124813f162eef52b3d81b51df63e2dc8166ad72da26ee88e5873b6b0fd9"}
Jul 17 05:35:05.587380 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:05.586932    2582 scope.go:117] "RemoveContainer" containerID="5a7b9f1d1ae37965974c5bd4fb706b0104f6f901f2ec5c0c7dfa6dabf47da659"
Jul 17 05:35:05.587380 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:05.586890    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"
Jul 17 05:35:05.607813 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:05.607789    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"]
Jul 17 05:35:05.613526 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:05.613502    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-c5pwc"]
Jul 17 05:35:06.729853 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:06.729821    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" path="/var/lib/kubelet/pods/04ed2466-fd0b-4d23-8dee-2966f03db083/volumes"
Jul 17 05:35:15.725333 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:15.725299    2582 scope.go:117] "RemoveContainer" containerID="b701ad873cc692f5d9dafc0ae92d036a195422c213e261b0d10dc3d158c4be2c"
Jul 17 05:35:16.635023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:16.634926    2582 generic.go:356] "Generic (PLEG): container finished" podID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerID="c0702408789e944942604e414bf23b4281a26b5a3d8bb8ef1aff1a49dea71a98" exitCode=6
Jul 17 05:35:16.635023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:16.635002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" event={"ID":"a009a53a-f9ec-4068-b394-41d25b11cd01","Type":"ContainerDied","Data":"c0702408789e944942604e414bf23b4281a26b5a3d8bb8ef1aff1a49dea71a98"}
Jul 17 05:35:16.635321 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:16.635080    2582 scope.go:117] "RemoveContainer" containerID="b701ad873cc692f5d9dafc0ae92d036a195422c213e261b0d10dc3d158c4be2c"
Jul 17 05:35:16.635524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:16.635502    2582 scope.go:117] "RemoveContainer" containerID="c0702408789e944942604e414bf23b4281a26b5a3d8bb8ef1aff1a49dea71a98"
Jul 17 05:35:16.635726 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:16.635701    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-r68jm_odh-ai-gateway-infra(a009a53a-f9ec-4068-b394-41d25b11cd01)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01"
Jul 17 05:35:17.203785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.203753    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"]
Jul 17 05:35:17.204158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.204036    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:35:17.204158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.204047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:35:17.204158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.204056    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:35:17.204158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.204061    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:35:17.204158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.204115    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:35:17.204158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.204123    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:35:17.208042 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.208025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:17.218038 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.218013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"]
Jul 17 05:35:17.313811 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.313777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44900d6b-6231-4296-9d13-82e5ac57d25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4\" (UID: \"44900d6b-6231-4296-9d13-82e5ac57d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:17.414974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.414939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44900d6b-6231-4296-9d13-82e5ac57d25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4\" (UID: \"44900d6b-6231-4296-9d13-82e5ac57d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:17.415585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.415566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44900d6b-6231-4296-9d13-82e5ac57d25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4\" (UID: \"44900d6b-6231-4296-9d13-82e5ac57d25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:17.518250 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.518161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:17.680403 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.680375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"]
Jul 17 05:35:17.681797 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:35:17.681765    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44900d6b_6231_4296_9d13_82e5ac57d25f.slice/crio-45d89271deae5cbb12309a653e7a65136b47f5ef6d709762dd18cb297128cc1f WatchSource:0}: Error finding container 45d89271deae5cbb12309a653e7a65136b47f5ef6d709762dd18cb297128cc1f: Status 404 returned error can't find the container with id 45d89271deae5cbb12309a653e7a65136b47f5ef6d709762dd18cb297128cc1f
Jul 17 05:35:17.700154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.700128    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"]
Jul 17 05:35:17.840207 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.840181    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:17.919990 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.919966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009a53a-f9ec-4068-b394-41d25b11cd01-maas-api-service-ca\") pod \"a009a53a-f9ec-4068-b394-41d25b11cd01\" (UID: \"a009a53a-f9ec-4068-b394-41d25b11cd01\") "
Jul 17 05:35:17.920304 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:17.920286    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a009a53a-f9ec-4068-b394-41d25b11cd01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a009a53a-f9ec-4068-b394-41d25b11cd01" (UID: "a009a53a-f9ec-4068-b394-41d25b11cd01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:35:18.020598 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.020567    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009a53a-f9ec-4068-b394-41d25b11cd01-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:35:18.647155 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.647062    2582 generic.go:356] "Generic (PLEG): container finished" podID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerID="c5cd3f25f492820881be3eaf3e2347c9071ae6de5f79fa45d3acb7c26d6a9dd4" exitCode=6
Jul 17 05:35:18.647622 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.647147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" event={"ID":"44900d6b-6231-4296-9d13-82e5ac57d25f","Type":"ContainerDied","Data":"c5cd3f25f492820881be3eaf3e2347c9071ae6de5f79fa45d3acb7c26d6a9dd4"}
Jul 17 05:35:18.647622 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.647202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" event={"ID":"44900d6b-6231-4296-9d13-82e5ac57d25f","Type":"ContainerStarted","Data":"45d89271deae5cbb12309a653e7a65136b47f5ef6d709762dd18cb297128cc1f"}
Jul 17 05:35:18.647622 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.647424    2582 scope.go:117] "RemoveContainer" containerID="c5cd3f25f492820881be3eaf3e2347c9071ae6de5f79fa45d3acb7c26d6a9dd4"
Jul 17 05:35:18.648518 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.648489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm" event={"ID":"a009a53a-f9ec-4068-b394-41d25b11cd01","Type":"ContainerDied","Data":"9880a1857eb56ada2fa445255d773772f5f9ee23f14f79095d612d33289e8db4"}
Jul 17 05:35:18.648518 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.648515    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"
Jul 17 05:35:18.648693 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.648535    2582 scope.go:117] "RemoveContainer" containerID="c0702408789e944942604e414bf23b4281a26b5a3d8bb8ef1aff1a49dea71a98"
Jul 17 05:35:18.676551 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.676525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"]
Jul 17 05:35:18.682224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.682202    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-r68jm"]
Jul 17 05:35:18.729654 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:18.729629    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" path="/var/lib/kubelet/pods/a009a53a-f9ec-4068-b394-41d25b11cd01/volumes"
Jul 17 05:35:19.655786 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:19.655706    2582 generic.go:356] "Generic (PLEG): container finished" podID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerID="6e3ce34a6541d9c3cec8b214c663e9b5d099fc746f8173e29e0b7eab0adc96af" exitCode=6
Jul 17 05:35:19.655786 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:19.655756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" event={"ID":"44900d6b-6231-4296-9d13-82e5ac57d25f","Type":"ContainerDied","Data":"6e3ce34a6541d9c3cec8b214c663e9b5d099fc746f8173e29e0b7eab0adc96af"}
Jul 17 05:35:19.656229 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:19.655806    2582 scope.go:117] "RemoveContainer" containerID="c5cd3f25f492820881be3eaf3e2347c9071ae6de5f79fa45d3acb7c26d6a9dd4"
Jul 17 05:35:19.656229 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:19.656008    2582 scope.go:117] "RemoveContainer" containerID="6e3ce34a6541d9c3cec8b214c663e9b5d099fc746f8173e29e0b7eab0adc96af"
Jul 17 05:35:19.656359 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:19.656251    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4_odh-ai-gateway-infra(44900d6b-6231-4296-9d13-82e5ac57d25f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f"
Jul 17 05:35:20.672146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:20.672120    2582 scope.go:117] "RemoveContainer" containerID="6e3ce34a6541d9c3cec8b214c663e9b5d099fc746f8173e29e0b7eab0adc96af"
Jul 17 05:35:20.672559 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:20.672313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4_odh-ai-gateway-infra(44900d6b-6231-4296-9d13-82e5ac57d25f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f"
Jul 17 05:35:31.725211 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:31.725165    2582 scope.go:117] "RemoveContainer" containerID="6e3ce34a6541d9c3cec8b214c663e9b5d099fc746f8173e29e0b7eab0adc96af"
Jul 17 05:35:32.722128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:32.722098    2582 generic.go:356] "Generic (PLEG): container finished" podID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerID="d55e37c1a207469af03ec8cb1dc17f2e84b616d70cd45e7d3a7e18f90b512aea" exitCode=6
Jul 17 05:35:32.722376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:32.722174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" event={"ID":"44900d6b-6231-4296-9d13-82e5ac57d25f","Type":"ContainerDied","Data":"d55e37c1a207469af03ec8cb1dc17f2e84b616d70cd45e7d3a7e18f90b512aea"}
Jul 17 05:35:32.722376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:32.722241    2582 scope.go:117] "RemoveContainer" containerID="6e3ce34a6541d9c3cec8b214c663e9b5d099fc746f8173e29e0b7eab0adc96af"
Jul 17 05:35:32.722538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:32.722522    2582 scope.go:117] "RemoveContainer" containerID="d55e37c1a207469af03ec8cb1dc17f2e84b616d70cd45e7d3a7e18f90b512aea"
Jul 17 05:35:32.722732 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:35:32.722713    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4_odh-ai-gateway-infra(44900d6b-6231-4296-9d13-82e5ac57d25f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f"
Jul 17 05:35:33.750188 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:33.750149    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"]
Jul 17 05:35:33.878544 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:33.878519    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:33.953303 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:33.953246    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44900d6b-6231-4296-9d13-82e5ac57d25f-maas-api-service-ca\") pod \"44900d6b-6231-4296-9d13-82e5ac57d25f\" (UID: \"44900d6b-6231-4296-9d13-82e5ac57d25f\") "
Jul 17 05:35:33.953640 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:33.953614    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44900d6b-6231-4296-9d13-82e5ac57d25f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44900d6b-6231-4296-9d13-82e5ac57d25f" (UID: "44900d6b-6231-4296-9d13-82e5ac57d25f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:35:34.054817 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:34.054735    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44900d6b-6231-4296-9d13-82e5ac57d25f-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:35:34.732924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:34.732892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4" event={"ID":"44900d6b-6231-4296-9d13-82e5ac57d25f","Type":"ContainerDied","Data":"45d89271deae5cbb12309a653e7a65136b47f5ef6d709762dd18cb297128cc1f"}
Jul 17 05:35:34.732924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:34.732924    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"
Jul 17 05:35:34.733289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:34.732952    2582 scope.go:117] "RemoveContainer" containerID="d55e37c1a207469af03ec8cb1dc17f2e84b616d70cd45e7d3a7e18f90b512aea"
Jul 17 05:35:34.756801 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:34.756172    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"]
Jul 17 05:35:34.761842 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:34.761819    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-lrmt4"]
Jul 17 05:35:36.729942 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:35:36.729907    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" path="/var/lib/kubelet/pods/44900d6b-6231-4296-9d13-82e5ac57d25f/volumes"
Jul 17 05:36:29.198337 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198303    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"]
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198581    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198591    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198608    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198621    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198626    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198634    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198639    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198647    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198651    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198660    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198709    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198717    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198723    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198728    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44900d6b-6231-4296-9d13-82e5ac57d25f" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198734    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:29.198849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.198739    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ed2466-fd0b-4d23-8dee-2966f03db083" containerName="revoke-keys"
Jul 17 05:36:29.201542 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.201525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:29.204455 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.204429    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:36:29.204562 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.204439    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:36:29.209071 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.208969    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"]
Jul 17 05:36:29.297179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.297137    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6284e903-c3e5-4230-ae98-4a21a210ad6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7\" (UID: \"6284e903-c3e5-4230-ae98-4a21a210ad6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:29.397611 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.397577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6284e903-c3e5-4230-ae98-4a21a210ad6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7\" (UID: \"6284e903-c3e5-4230-ae98-4a21a210ad6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:29.398245 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.398227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6284e903-c3e5-4230-ae98-4a21a210ad6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7\" (UID: \"6284e903-c3e5-4230-ae98-4a21a210ad6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:29.513429 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.513331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:29.638757 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.638733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"]
Jul 17 05:36:29.641368 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:36:29.641342    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6284e903_c3e5_4230_ae98_4a21a210ad6d.slice/crio-aebf091a68a2da69fdc1f47fc526ee1610e28729c1a7781037f5c03851a0e173 WatchSource:0}: Error finding container aebf091a68a2da69fdc1f47fc526ee1610e28729c1a7781037f5c03851a0e173: Status 404 returned error can't find the container with id aebf091a68a2da69fdc1f47fc526ee1610e28729c1a7781037f5c03851a0e173
Jul 17 05:36:29.964740 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.964702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerStarted","Data":"ad3b88b3cfd17cf12fb5b981c74504d6d49a0849c9452a87f0614d1c048284c2"}
Jul 17 05:36:29.964918 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.964755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerStarted","Data":"aebf091a68a2da69fdc1f47fc526ee1610e28729c1a7781037f5c03851a0e173"}
Jul 17 05:36:29.980081 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:29.980033    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" podStartSLOduration=0.980016333 podStartE2EDuration="980.016333ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:36:29.979554032 +0000 UTC m=+2793.716616371" watchObservedRunningTime="2026-07-17 05:36:29.980016333 +0000 UTC m=+2793.717078670"
Jul 17 05:36:30.977658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:30.977604    2582 generic.go:356] "Generic (PLEG): container finished" podID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerID="ad3b88b3cfd17cf12fb5b981c74504d6d49a0849c9452a87f0614d1c048284c2" exitCode=6
Jul 17 05:36:30.978147 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:30.977741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerDied","Data":"ad3b88b3cfd17cf12fb5b981c74504d6d49a0849c9452a87f0614d1c048284c2"}
Jul 17 05:36:30.978223 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:30.978170    2582 scope.go:117] "RemoveContainer" containerID="ad3b88b3cfd17cf12fb5b981c74504d6d49a0849c9452a87f0614d1c048284c2"
Jul 17 05:36:31.983940 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:31.983905    2582 generic.go:356] "Generic (PLEG): container finished" podID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerID="aff09ae9477f6b66a915fec8d26f454360e9e83837e02c381dab42222da13ef1" exitCode=6
Jul 17 05:36:31.984345 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:31.983993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerDied","Data":"aff09ae9477f6b66a915fec8d26f454360e9e83837e02c381dab42222da13ef1"}
Jul 17 05:36:31.984345 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:31.984054    2582 scope.go:117] "RemoveContainer" containerID="ad3b88b3cfd17cf12fb5b981c74504d6d49a0849c9452a87f0614d1c048284c2"
Jul 17 05:36:31.984345 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:31.984308    2582 scope.go:117] "RemoveContainer" containerID="aff09ae9477f6b66a915fec8d26f454360e9e83837e02c381dab42222da13ef1"
Jul 17 05:36:31.984518 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:36:31.984500    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7_odh-ai-gateway-infra(6284e903-c3e5-4230-ae98-4a21a210ad6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d"
Jul 17 05:36:32.989606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:32.989574    2582 scope.go:117] "RemoveContainer" containerID="aff09ae9477f6b66a915fec8d26f454360e9e83837e02c381dab42222da13ef1"
Jul 17 05:36:32.990081 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:36:32.989803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7_odh-ai-gateway-infra(6284e903-c3e5-4230-ae98-4a21a210ad6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d"
Jul 17 05:36:33.229569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.229535    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"]
Jul 17 05:36:33.229849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.229837    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:33.229905 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.229851    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:33.229943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.229911    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a009a53a-f9ec-4068-b394-41d25b11cd01" containerName="revoke-keys"
Jul 17 05:36:33.233737 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.233718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.237550 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.237521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e1ab5d\"/\"kube-root-ca.crt\""
Jul 17 05:36:33.237550 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.237538    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e1ab5d\"/\"test-model-e1ab5d-kserve-self-signed-certs\""
Jul 17 05:36:33.237732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.237573    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e1ab5d\"/\"default-dockercfg-6h25v\""
Jul 17 05:36:33.237732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.237623    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e1ab5d\"/\"openshift-service-ca.crt\""
Jul 17 05:36:33.244910 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.244883    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"]
Jul 17 05:36:33.332532 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.332492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-dshm\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.332711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.332547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-model-cache\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.332711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.332573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tmp-dir\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.332711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.332688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tls-certs\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.332830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.332731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-home\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.332830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.332756    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7829x\" (UniqueName: \"kubernetes.io/projected/10d39eba-1207-4069-a1b5-d7c0c6382fa2-kube-api-access-7829x\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.433773 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.433734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-model-cache\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.433773 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.433776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tmp-dir\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434011 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.433816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tls-certs\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434052 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.434006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-home\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.434065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7829x\" (UniqueName: \"kubernetes.io/projected/10d39eba-1207-4069-a1b5-d7c0c6382fa2-kube-api-access-7829x\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434127 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.434104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-dshm\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434219 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.434198    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tmp-dir\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434302 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.434229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-model-cache\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.434351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.434319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-home\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.436422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.436401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-dshm\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.436564 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.436546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tls-certs\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.444514 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.444485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7829x\" (UniqueName: \"kubernetes.io/projected/10d39eba-1207-4069-a1b5-d7c0c6382fa2-kube-api-access-7829x\") pod \"test-model-e1ab5d-kserve-78c5f7bb49-wxbv4\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") " pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.542638 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.542530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:33.676303 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.676197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"]
Jul 17 05:36:33.678577 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:36:33.678546    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10d39eba_1207_4069_a1b5_d7c0c6382fa2.slice/crio-989ef8af1a3a07a07ebf4f20d9eb7d902b1cd405e4687bd031be9fec7ac9f372 WatchSource:0}: Error finding container 989ef8af1a3a07a07ebf4f20d9eb7d902b1cd405e4687bd031be9fec7ac9f372: Status 404 returned error can't find the container with id 989ef8af1a3a07a07ebf4f20d9eb7d902b1cd405e4687bd031be9fec7ac9f372
Jul 17 05:36:33.995138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.995095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4" event={"ID":"10d39eba-1207-4069-a1b5-d7c0c6382fa2","Type":"ContainerStarted","Data":"a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3"}
Jul 17 05:36:33.995659 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:33.995147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4" event={"ID":"10d39eba-1207-4069-a1b5-d7c0c6382fa2","Type":"ContainerStarted","Data":"989ef8af1a3a07a07ebf4f20d9eb7d902b1cd405e4687bd031be9fec7ac9f372"}
Jul 17 05:36:34.014825 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:34.014770    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4" podStartSLOduration=1.014752649 podStartE2EDuration="1.014752649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:36:34.012928698 +0000 UTC m=+2797.749991086" watchObservedRunningTime="2026-07-17 05:36:34.014752649 +0000 UTC m=+2797.751815007"
Jul 17 05:36:43.542828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:43.542782    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:43.543311 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:43.542862    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:43.558870 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:43.558846    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:44.052037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:44.052003    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:36:47.724930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:47.724893    2582 scope.go:117] "RemoveContainer" containerID="aff09ae9477f6b66a915fec8d26f454360e9e83837e02c381dab42222da13ef1"
Jul 17 05:36:48.057429 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:48.057394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerStarted","Data":"8f8f37bc35bb384f9a704212b6d179e0d4cbab7ead2a128f436b97f20cf297ed"}
Jul 17 05:36:49.063763 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:49.063731    2582 generic.go:356] "Generic (PLEG): container finished" podID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerID="8f8f37bc35bb384f9a704212b6d179e0d4cbab7ead2a128f436b97f20cf297ed" exitCode=6
Jul 17 05:36:49.064199 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:49.063808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerDied","Data":"8f8f37bc35bb384f9a704212b6d179e0d4cbab7ead2a128f436b97f20cf297ed"}
Jul 17 05:36:49.064199 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:49.063866    2582 scope.go:117] "RemoveContainer" containerID="aff09ae9477f6b66a915fec8d26f454360e9e83837e02c381dab42222da13ef1"
Jul 17 05:36:49.064199 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:49.064183    2582 scope.go:117] "RemoveContainer" containerID="8f8f37bc35bb384f9a704212b6d179e0d4cbab7ead2a128f436b97f20cf297ed"
Jul 17 05:36:49.064464 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:36:49.064444    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7_odh-ai-gateway-infra(6284e903-c3e5-4230-ae98-4a21a210ad6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d"
Jul 17 05:36:49.085250 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:49.085223    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"]
Jul 17 05:36:50.201154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:50.201128    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:50.279360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:50.279325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6284e903-c3e5-4230-ae98-4a21a210ad6d-maas-api-service-ca\") pod \"6284e903-c3e5-4230-ae98-4a21a210ad6d\" (UID: \"6284e903-c3e5-4230-ae98-4a21a210ad6d\") "
Jul 17 05:36:50.279728 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:50.279706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6284e903-c3e5-4230-ae98-4a21a210ad6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6284e903-c3e5-4230-ae98-4a21a210ad6d" (UID: "6284e903-c3e5-4230-ae98-4a21a210ad6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:36:50.380901 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:50.380815    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6284e903-c3e5-4230-ae98-4a21a210ad6d-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:36:51.075055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.074947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7" event={"ID":"6284e903-c3e5-4230-ae98-4a21a210ad6d","Type":"ContainerDied","Data":"aebf091a68a2da69fdc1f47fc526ee1610e28729c1a7781037f5c03851a0e173"}
Jul 17 05:36:51.075055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.075026    2582 scope.go:117] "RemoveContainer" containerID="8f8f37bc35bb384f9a704212b6d179e0d4cbab7ead2a128f436b97f20cf297ed"
Jul 17 05:36:51.075319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.075129    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"
Jul 17 05:36:51.086405 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086379    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"]
Jul 17 05:36:51.086664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086652    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.086705 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.086705 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086682    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.086705 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086688    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.086803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086744    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.086803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086751    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.086803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.086758    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:36:51.090913 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.090897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:36:51.093606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.093581    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:36:51.093884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.093863    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:36:51.095436 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.095412    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"]
Jul 17 05:36:51.098320 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.098298    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"]
Jul 17 05:36:51.101456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.101433    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-j4nh7"]
Jul 17 05:36:51.189086 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.189043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-8hpb2\" (UID: \"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:36:51.289544 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.289501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-8hpb2\" (UID: \"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:36:51.290115 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.290096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-8hpb2\" (UID: \"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:36:51.402883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.402800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:36:51.532887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:51.532862    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"]
Jul 17 05:36:51.535515 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:36:51.535489    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2486db7e_2fc7_4b49_80ba_b9d75ecf2f07.slice/crio-aefb8a01e920225730e0b3e794c92b164c240cebbc8f2f840bb368df10a7c180 WatchSource:0}: Error finding container aefb8a01e920225730e0b3e794c92b164c240cebbc8f2f840bb368df10a7c180: Status 404 returned error can't find the container with id aefb8a01e920225730e0b3e794c92b164c240cebbc8f2f840bb368df10a7c180
Jul 17 05:36:52.080771 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:52.080677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerStarted","Data":"aca5fa66cd9d1ce83d0538e255680b417a909a5fb68de8e2571a90b05c33383e"}
Jul 17 05:36:52.080771 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:52.080737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerStarted","Data":"aefb8a01e920225730e0b3e794c92b164c240cebbc8f2f840bb368df10a7c180"}
Jul 17 05:36:52.097849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:52.097715    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" podStartSLOduration=1.09769685 podStartE2EDuration="1.09769685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:36:52.097051824 +0000 UTC m=+2815.834114160" watchObservedRunningTime="2026-07-17 05:36:52.09769685 +0000 UTC m=+2815.834759188"
Jul 17 05:36:52.730323 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:52.730248    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" path="/var/lib/kubelet/pods/6284e903-c3e5-4230-ae98-4a21a210ad6d/volumes"
Jul 17 05:36:53.087566 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:53.087471    2582 generic.go:356] "Generic (PLEG): container finished" podID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerID="aca5fa66cd9d1ce83d0538e255680b417a909a5fb68de8e2571a90b05c33383e" exitCode=6
Jul 17 05:36:53.087732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:53.087564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerDied","Data":"aca5fa66cd9d1ce83d0538e255680b417a909a5fb68de8e2571a90b05c33383e"}
Jul 17 05:36:53.087787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:53.087729    2582 scope.go:117] "RemoveContainer" containerID="aca5fa66cd9d1ce83d0538e255680b417a909a5fb68de8e2571a90b05c33383e"
Jul 17 05:36:54.096556 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:54.096522    2582 generic.go:356] "Generic (PLEG): container finished" podID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerID="24f6b1de843a3f32385377b120b65023538e63c9cef7a121fd795d4495b14f77" exitCode=6
Jul 17 05:36:54.097004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:54.096578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerDied","Data":"24f6b1de843a3f32385377b120b65023538e63c9cef7a121fd795d4495b14f77"}
Jul 17 05:36:54.097004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:54.096638    2582 scope.go:117] "RemoveContainer" containerID="aca5fa66cd9d1ce83d0538e255680b417a909a5fb68de8e2571a90b05c33383e"
Jul 17 05:36:54.097004 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:54.096824    2582 scope.go:117] "RemoveContainer" containerID="24f6b1de843a3f32385377b120b65023538e63c9cef7a121fd795d4495b14f77"
Jul 17 05:36:54.097126 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:36:54.097041    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-8hpb2_odh-ai-gateway-infra(2486db7e-2fc7-4b49-80ba-b9d75ecf2f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07"
Jul 17 05:36:55.102493 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:36:55.102464    2582 scope.go:117] "RemoveContainer" containerID="24f6b1de843a3f32385377b120b65023538e63c9cef7a121fd795d4495b14f77"
Jul 17 05:36:55.102883 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:36:55.102679    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-8hpb2_odh-ai-gateway-infra(2486db7e-2fc7-4b49-80ba-b9d75ecf2f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07"
Jul 17 05:37:07.291185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.291152    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"]
Jul 17 05:37:07.291688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.291465    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:37:07.291688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.291476    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6284e903-c3e5-4230-ae98-4a21a210ad6d" containerName="revoke-keys"
Jul 17 05:37:07.294326 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.294308    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:07.306330 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.302329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"]
Jul 17 05:37:07.420787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.420752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33edf2a3-2abf-4304-819b-3e867823be1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth\" (UID: \"33edf2a3-2abf-4304-819b-3e867823be1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:07.522111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.522066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33edf2a3-2abf-4304-819b-3e867823be1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth\" (UID: \"33edf2a3-2abf-4304-819b-3e867823be1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:07.522736 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.522716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33edf2a3-2abf-4304-819b-3e867823be1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth\" (UID: \"33edf2a3-2abf-4304-819b-3e867823be1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:07.608309 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.608195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:07.740395 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:07.740330    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"]
Jul 17 05:37:07.742755 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:37:07.742730    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33edf2a3_2abf_4304_819b_3e867823be1a.slice/crio-03f9b61499236c3dce519d74cc3f23249dafe3f1f58a8777ae58ef79bf3f13d2 WatchSource:0}: Error finding container 03f9b61499236c3dce519d74cc3f23249dafe3f1f58a8777ae58ef79bf3f13d2: Status 404 returned error can't find the container with id 03f9b61499236c3dce519d74cc3f23249dafe3f1f58a8777ae58ef79bf3f13d2
Jul 17 05:37:08.158284 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:08.158167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerStarted","Data":"661de64ae7230dc3a61642cfd0dc96adf60bf6484491b2db435f1ca499e317f6"}
Jul 17 05:37:08.158284 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:08.158221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerStarted","Data":"03f9b61499236c3dce519d74cc3f23249dafe3f1f58a8777ae58ef79bf3f13d2"}
Jul 17 05:37:08.176461 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:08.176418    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" podStartSLOduration=1.176401953 podStartE2EDuration="1.176401953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:37:08.173991786 +0000 UTC m=+2831.911054121" watchObservedRunningTime="2026-07-17 05:37:08.176401953 +0000 UTC m=+2831.913464288"
Jul 17 05:37:09.164365 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:09.164332    2582 generic.go:356] "Generic (PLEG): container finished" podID="33edf2a3-2abf-4304-819b-3e867823be1a" containerID="661de64ae7230dc3a61642cfd0dc96adf60bf6484491b2db435f1ca499e317f6" exitCode=6
Jul 17 05:37:09.164830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:09.164409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerDied","Data":"661de64ae7230dc3a61642cfd0dc96adf60bf6484491b2db435f1ca499e317f6"}
Jul 17 05:37:09.164830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:09.164603    2582 scope.go:117] "RemoveContainer" containerID="661de64ae7230dc3a61642cfd0dc96adf60bf6484491b2db435f1ca499e317f6"
Jul 17 05:37:09.725221 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:09.725186    2582 scope.go:117] "RemoveContainer" containerID="24f6b1de843a3f32385377b120b65023538e63c9cef7a121fd795d4495b14f77"
Jul 17 05:37:10.170455 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:10.170360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerStarted","Data":"69a320a2c1114c9e8466ff9fcc6e783f5dd40a80ab8d8434b642da9842dc39d5"}
Jul 17 05:37:10.172113 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:10.172089    2582 generic.go:356] "Generic (PLEG): container finished" podID="33edf2a3-2abf-4304-819b-3e867823be1a" containerID="a8099f63a36bd815a6c5bc52ab95eba8e139d3ccf5887894cae15decd869a97f" exitCode=6
Jul 17 05:37:10.172239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:10.172150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerDied","Data":"a8099f63a36bd815a6c5bc52ab95eba8e139d3ccf5887894cae15decd869a97f"}
Jul 17 05:37:10.172239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:10.172198    2582 scope.go:117] "RemoveContainer" containerID="661de64ae7230dc3a61642cfd0dc96adf60bf6484491b2db435f1ca499e317f6"
Jul 17 05:37:10.172376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:10.172360    2582 scope.go:117] "RemoveContainer" containerID="a8099f63a36bd815a6c5bc52ab95eba8e139d3ccf5887894cae15decd869a97f"
Jul 17 05:37:10.172567 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:10.172550    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth_odh-ai-gateway-infra(33edf2a3-2abf-4304-819b-3e867823be1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" podUID="33edf2a3-2abf-4304-819b-3e867823be1a"
Jul 17 05:37:11.178485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:11.178450    2582 generic.go:356] "Generic (PLEG): container finished" podID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerID="69a320a2c1114c9e8466ff9fcc6e783f5dd40a80ab8d8434b642da9842dc39d5" exitCode=6
Jul 17 05:37:11.178950 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:11.178525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerDied","Data":"69a320a2c1114c9e8466ff9fcc6e783f5dd40a80ab8d8434b642da9842dc39d5"}
Jul 17 05:37:11.178950 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:11.178579    2582 scope.go:117] "RemoveContainer" containerID="24f6b1de843a3f32385377b120b65023538e63c9cef7a121fd795d4495b14f77"
Jul 17 05:37:11.178950 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:11.178911    2582 scope.go:117] "RemoveContainer" containerID="69a320a2c1114c9e8466ff9fcc6e783f5dd40a80ab8d8434b642da9842dc39d5"
Jul 17 05:37:11.179147 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:11.179126    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-8hpb2_odh-ai-gateway-infra(2486db7e-2fc7-4b49-80ba-b9d75ecf2f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07"
Jul 17 05:37:11.180366 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:11.180350    2582 scope.go:117] "RemoveContainer" containerID="a8099f63a36bd815a6c5bc52ab95eba8e139d3ccf5887894cae15decd869a97f"
Jul 17 05:37:11.180585 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:11.180557    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth_odh-ai-gateway-infra(33edf2a3-2abf-4304-819b-3e867823be1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" podUID="33edf2a3-2abf-4304-819b-3e867823be1a"
Jul 17 05:37:11.197838 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:11.197812    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"]
Jul 17 05:37:12.315360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:12.315337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:37:12.359843 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:12.359812    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07-maas-api-service-ca\") pod \"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07\" (UID: \"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07\") "
Jul 17 05:37:12.360143 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:12.360122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" (UID: "2486db7e-2fc7-4b49-80ba-b9d75ecf2f07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:37:12.460670 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:12.460635    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:13.190894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:13.190865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2" event={"ID":"2486db7e-2fc7-4b49-80ba-b9d75ecf2f07","Type":"ContainerDied","Data":"aefb8a01e920225730e0b3e794c92b164c240cebbc8f2f840bb368df10a7c180"}
Jul 17 05:37:13.191039 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:13.190899    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"
Jul 17 05:37:13.191039 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:13.190933    2582 scope.go:117] "RemoveContainer" containerID="69a320a2c1114c9e8466ff9fcc6e783f5dd40a80ab8d8434b642da9842dc39d5"
Jul 17 05:37:13.207160 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:13.207135    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"]
Jul 17 05:37:13.215414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:13.215392    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-8hpb2"]
Jul 17 05:37:14.729841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:14.729808    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" path="/var/lib/kubelet/pods/2486db7e-2fc7-4b49-80ba-b9d75ecf2f07/volumes"
Jul 17 05:37:23.725178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:23.725148    2582 scope.go:117] "RemoveContainer" containerID="a8099f63a36bd815a6c5bc52ab95eba8e139d3ccf5887894cae15decd869a97f"
Jul 17 05:37:24.242686 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:24.242604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerStarted","Data":"7ef55ff16f88000da37e9b62ba10d4afb7d5ad47722d608074fcc801b73efd41"}
Jul 17 05:37:25.028391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:25.028355    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"]
Jul 17 05:37:25.249121 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:25.249087    2582 generic.go:356] "Generic (PLEG): container finished" podID="33edf2a3-2abf-4304-819b-3e867823be1a" containerID="7ef55ff16f88000da37e9b62ba10d4afb7d5ad47722d608074fcc801b73efd41" exitCode=6
Jul 17 05:37:25.249313 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:25.249159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerDied","Data":"7ef55ff16f88000da37e9b62ba10d4afb7d5ad47722d608074fcc801b73efd41"}
Jul 17 05:37:25.249313 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:25.249219    2582 scope.go:117] "RemoveContainer" containerID="a8099f63a36bd815a6c5bc52ab95eba8e139d3ccf5887894cae15decd869a97f"
Jul 17 05:37:25.249547 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:25.249522    2582 scope.go:117] "RemoveContainer" containerID="7ef55ff16f88000da37e9b62ba10d4afb7d5ad47722d608074fcc801b73efd41"
Jul 17 05:37:25.249772 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:25.249753    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth_odh-ai-gateway-infra(33edf2a3-2abf-4304-819b-3e867823be1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" podUID="33edf2a3-2abf-4304-819b-3e867823be1a"
Jul 17 05:37:25.276935 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:25.276901    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"]
Jul 17 05:37:26.087949 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.087907    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"]
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088268    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088280    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088305    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088314    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088363    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088373    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.088389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.088381    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:26.092493 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.092473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:26.099748 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.099667    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"]
Jul 17 05:37:26.169131 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.169090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d3aa4a-f08a-4822-89be-dad8af281b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns\" (UID: \"85d3aa4a-f08a-4822-89be-dad8af281b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:26.269908 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.269878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d3aa4a-f08a-4822-89be-dad8af281b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns\" (UID: \"85d3aa4a-f08a-4822-89be-dad8af281b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:26.270510 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.270492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d3aa4a-f08a-4822-89be-dad8af281b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns\" (UID: \"85d3aa4a-f08a-4822-89be-dad8af281b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:26.384137 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.384110    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:26.404390 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.404356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:26.471333 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.471138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33edf2a3-2abf-4304-819b-3e867823be1a-maas-api-service-ca\") pod \"33edf2a3-2abf-4304-819b-3e867823be1a\" (UID: \"33edf2a3-2abf-4304-819b-3e867823be1a\") "
Jul 17 05:37:26.471802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.471767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33edf2a3-2abf-4304-819b-3e867823be1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33edf2a3-2abf-4304-819b-3e867823be1a" (UID: "33edf2a3-2abf-4304-819b-3e867823be1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:37:26.542583 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.542557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"]
Jul 17 05:37:26.544863 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:37:26.544830    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d3aa4a_f08a_4822_89be_dad8af281b01.slice/crio-20266d6e5baaa4a84d90eedb52eac3c061a73aeb0d85281ab571c17261c7507c WatchSource:0}: Error finding container 20266d6e5baaa4a84d90eedb52eac3c061a73aeb0d85281ab571c17261c7507c: Status 404 returned error can't find the container with id 20266d6e5baaa4a84d90eedb52eac3c061a73aeb0d85281ab571c17261c7507c
Jul 17 05:37:26.572477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:26.572443    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33edf2a3-2abf-4304-819b-3e867823be1a-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:27.261366 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.261328    2582 generic.go:356] "Generic (PLEG): container finished" podID="85d3aa4a-f08a-4822-89be-dad8af281b01" containerID="5c11f86af7b7f3ca0ab1ceb3fbc9338bd46f28ff098b73e7c8d5ba5467e394f9" exitCode=0
Jul 17 05:37:27.261836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.261409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns" event={"ID":"85d3aa4a-f08a-4822-89be-dad8af281b01","Type":"ContainerDied","Data":"5c11f86af7b7f3ca0ab1ceb3fbc9338bd46f28ff098b73e7c8d5ba5467e394f9"}
Jul 17 05:37:27.261836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.261472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns" event={"ID":"85d3aa4a-f08a-4822-89be-dad8af281b01","Type":"ContainerStarted","Data":"20266d6e5baaa4a84d90eedb52eac3c061a73aeb0d85281ab571c17261c7507c"}
Jul 17 05:37:27.262695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.262671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth" event={"ID":"33edf2a3-2abf-4304-819b-3e867823be1a","Type":"ContainerDied","Data":"03f9b61499236c3dce519d74cc3f23249dafe3f1f58a8777ae58ef79bf3f13d2"}
Jul 17 05:37:27.262776 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.262693    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"
Jul 17 05:37:27.262776 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.262724    2582 scope.go:117] "RemoveContainer" containerID="7ef55ff16f88000da37e9b62ba10d4afb7d5ad47722d608074fcc801b73efd41"
Jul 17 05:37:27.290184 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.290157    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"]
Jul 17 05:37:27.296944 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:27.296918    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-gkqth"]
Jul 17 05:37:28.393550 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:28.393529    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:28.488671 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:28.488642    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d3aa4a-f08a-4822-89be-dad8af281b01-maas-api-service-ca\") pod \"85d3aa4a-f08a-4822-89be-dad8af281b01\" (UID: \"85d3aa4a-f08a-4822-89be-dad8af281b01\") "
Jul 17 05:37:28.488991 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:28.488969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d3aa4a-f08a-4822-89be-dad8af281b01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85d3aa4a-f08a-4822-89be-dad8af281b01" (UID: "85d3aa4a-f08a-4822-89be-dad8af281b01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:37:28.589399 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:28.589316    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85d3aa4a-f08a-4822-89be-dad8af281b01-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:28.730387 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:28.730355    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" path="/var/lib/kubelet/pods/33edf2a3-2abf-4304-819b-3e867823be1a/volumes"
Jul 17 05:37:29.250254 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250218    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"]
Jul 17 05:37:29.250539 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250526    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250541    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250555    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250561    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250570    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d3aa4a-f08a-4822-89be-dad8af281b01" containerName="revoke-keys"
Jul 17 05:37:29.250585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250576    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d3aa4a-f08a-4822-89be-dad8af281b01" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250588    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250593    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250599    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250605    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2486db7e-2fc7-4b49-80ba-b9d75ecf2f07" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250649    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250660    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d3aa4a-f08a-4822-89be-dad8af281b01" containerName="revoke-keys"
Jul 17 05:37:29.250746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.250666    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:29.253671 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.253652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:29.260605 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.260580    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"]
Jul 17 05:37:29.274251 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.274220    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns"
Jul 17 05:37:29.274441 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.274302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns" event={"ID":"85d3aa4a-f08a-4822-89be-dad8af281b01","Type":"ContainerDied","Data":"20266d6e5baaa4a84d90eedb52eac3c061a73aeb0d85281ab571c17261c7507c"}
Jul 17 05:37:29.274441 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.274344    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20266d6e5baaa4a84d90eedb52eac3c061a73aeb0d85281ab571c17261c7507c"
Jul 17 05:37:29.294917 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.294879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf3017-b08b-469c-a21a-6993e9f5b93d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj\" (UID: \"ffbf3017-b08b-469c-a21a-6993e9f5b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:29.396251 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.396213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf3017-b08b-469c-a21a-6993e9f5b93d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj\" (UID: \"ffbf3017-b08b-469c-a21a-6993e9f5b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:29.396983 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.396959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf3017-b08b-469c-a21a-6993e9f5b93d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj\" (UID: \"ffbf3017-b08b-469c-a21a-6993e9f5b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:29.564791 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.564677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:29.691227 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:29.691202    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"]
Jul 17 05:37:29.693466 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:37:29.693440    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbf3017_b08b_469c_a21a_6993e9f5b93d.slice/crio-7bf5d0bb4c8938e2dc4d3f0b2a416b07de7bef62712af44ca6d7e218f55710d4 WatchSource:0}: Error finding container 7bf5d0bb4c8938e2dc4d3f0b2a416b07de7bef62712af44ca6d7e218f55710d4: Status 404 returned error can't find the container with id 7bf5d0bb4c8938e2dc4d3f0b2a416b07de7bef62712af44ca6d7e218f55710d4
Jul 17 05:37:30.279580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:30.279546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" event={"ID":"ffbf3017-b08b-469c-a21a-6993e9f5b93d","Type":"ContainerStarted","Data":"13e6d1cf80b7fa3a5cdb2c81d40981f56f9d1c2625e82ff1cc375eab69563a2f"}
Jul 17 05:37:30.279788 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:30.279600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" event={"ID":"ffbf3017-b08b-469c-a21a-6993e9f5b93d","Type":"ContainerStarted","Data":"7bf5d0bb4c8938e2dc4d3f0b2a416b07de7bef62712af44ca6d7e218f55710d4"}
Jul 17 05:37:30.296452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:30.296347    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" podStartSLOduration=1.296324114 podStartE2EDuration="1.296324114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:37:30.295340535 +0000 UTC m=+2854.032402897" watchObservedRunningTime="2026-07-17 05:37:30.296324114 +0000 UTC m=+2854.033386451"
Jul 17 05:37:31.285692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:31.285658    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerID="13e6d1cf80b7fa3a5cdb2c81d40981f56f9d1c2625e82ff1cc375eab69563a2f" exitCode=6
Jul 17 05:37:31.286117 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:31.285726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" event={"ID":"ffbf3017-b08b-469c-a21a-6993e9f5b93d","Type":"ContainerDied","Data":"13e6d1cf80b7fa3a5cdb2c81d40981f56f9d1c2625e82ff1cc375eab69563a2f"}
Jul 17 05:37:31.286117 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:31.285953    2582 scope.go:117] "RemoveContainer" containerID="13e6d1cf80b7fa3a5cdb2c81d40981f56f9d1c2625e82ff1cc375eab69563a2f"
Jul 17 05:37:32.291771 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:32.291738    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerID="6d9c723afee959e9a67b0dceb5b9b0bb376563aab7d9de6fd2d8fdb5065b640e" exitCode=6
Jul 17 05:37:32.292224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:32.291783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" event={"ID":"ffbf3017-b08b-469c-a21a-6993e9f5b93d","Type":"ContainerDied","Data":"6d9c723afee959e9a67b0dceb5b9b0bb376563aab7d9de6fd2d8fdb5065b640e"}
Jul 17 05:37:32.292224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:32.291862    2582 scope.go:117] "RemoveContainer" containerID="13e6d1cf80b7fa3a5cdb2c81d40981f56f9d1c2625e82ff1cc375eab69563a2f"
Jul 17 05:37:32.292224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:32.292005    2582 scope.go:117] "RemoveContainer" containerID="6d9c723afee959e9a67b0dceb5b9b0bb376563aab7d9de6fd2d8fdb5065b640e"
Jul 17 05:37:32.292400 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:32.292231    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj_odh-ai-gateway-infra(ffbf3017-b08b-469c-a21a-6993e9f5b93d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d"
Jul 17 05:37:33.298145 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:33.298107    2582 scope.go:117] "RemoveContainer" containerID="6d9c723afee959e9a67b0dceb5b9b0bb376563aab7d9de6fd2d8fdb5065b640e"
Jul 17 05:37:33.298549 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:33.298314    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj_odh-ai-gateway-infra(ffbf3017-b08b-469c-a21a-6993e9f5b93d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d"
Jul 17 05:37:34.677073 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.677039    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"]
Jul 17 05:37:34.677513 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.677415    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33edf2a3-2abf-4304-819b-3e867823be1a" containerName="revoke-keys"
Jul 17 05:37:34.680072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.680056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:34.687470 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.687446    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"]
Jul 17 05:37:34.745042 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.745007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z\" (UID: \"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:34.845431 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.845397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z\" (UID: \"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:34.845978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.845960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z\" (UID: \"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:34.991609 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:34.991568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:35.123447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:35.123418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"]
Jul 17 05:37:35.125061 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:37:35.125026    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c4d3b9_2d3c_49c2_8d09_596ef0761dc8.slice/crio-baee68abd53bcdbce8f4b3d9f3bd8e1d1f10ad7a5b5e0fcff17ff2173ed14413 WatchSource:0}: Error finding container baee68abd53bcdbce8f4b3d9f3bd8e1d1f10ad7a5b5e0fcff17ff2173ed14413: Status 404 returned error can't find the container with id baee68abd53bcdbce8f4b3d9f3bd8e1d1f10ad7a5b5e0fcff17ff2173ed14413
Jul 17 05:37:35.308505 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:35.308479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" event={"ID":"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8","Type":"ContainerStarted","Data":"baee68abd53bcdbce8f4b3d9f3bd8e1d1f10ad7a5b5e0fcff17ff2173ed14413"}
Jul 17 05:37:36.314141 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:36.314096    2582 generic.go:356] "Generic (PLEG): container finished" podID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerID="1ad5bf6097f395fe84ef7c44f40840db785eef6982ca7bfa32f1d1734443aa98" exitCode=6
Jul 17 05:37:36.314716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:36.314144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" event={"ID":"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8","Type":"ContainerDied","Data":"1ad5bf6097f395fe84ef7c44f40840db785eef6982ca7bfa32f1d1734443aa98"}
Jul 17 05:37:36.314716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:36.314453    2582 scope.go:117] "RemoveContainer" containerID="1ad5bf6097f395fe84ef7c44f40840db785eef6982ca7bfa32f1d1734443aa98"
Jul 17 05:37:37.320305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:37.320244    2582 generic.go:356] "Generic (PLEG): container finished" podID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerID="34afbee304ad6f79f4f9c40e7969e63882445bf9c727311c5f94767877dcd27b" exitCode=6
Jul 17 05:37:37.320789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:37.320334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" event={"ID":"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8","Type":"ContainerDied","Data":"34afbee304ad6f79f4f9c40e7969e63882445bf9c727311c5f94767877dcd27b"}
Jul 17 05:37:37.320789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:37.320411    2582 scope.go:117] "RemoveContainer" containerID="1ad5bf6097f395fe84ef7c44f40840db785eef6982ca7bfa32f1d1734443aa98"
Jul 17 05:37:37.320789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:37.320687    2582 scope.go:117] "RemoveContainer" containerID="34afbee304ad6f79f4f9c40e7969e63882445bf9c727311c5f94767877dcd27b"
Jul 17 05:37:37.320892 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:37.320871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z_odh-ai-gateway-infra(34c4d3b9-2d3c-49c2-8d09-596ef0761dc8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8"
Jul 17 05:37:38.326206 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:38.326181    2582 scope.go:117] "RemoveContainer" containerID="34afbee304ad6f79f4f9c40e7969e63882445bf9c727311c5f94767877dcd27b"
Jul 17 05:37:38.326665 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:38.326431    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z_odh-ai-gateway-infra(34c4d3b9-2d3c-49c2-8d09-596ef0761dc8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8"
Jul 17 05:37:40.048525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.048481    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4" podUID="10d39eba-1207-4069-a1b5-d7c0c6382fa2" containerName="main" containerID="cri-o://a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3" gracePeriod=45
Jul 17 05:37:40.297809 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.297784    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:37:40.338233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.338153    2582 generic.go:356] "Generic (PLEG): container finished" podID="10d39eba-1207-4069-a1b5-d7c0c6382fa2" containerID="a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3" exitCode=0
Jul 17 05:37:40.338233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.338199    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"
Jul 17 05:37:40.338485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.338235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4" event={"ID":"10d39eba-1207-4069-a1b5-d7c0c6382fa2","Type":"ContainerDied","Data":"a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3"}
Jul 17 05:37:40.338485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.338296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4" event={"ID":"10d39eba-1207-4069-a1b5-d7c0c6382fa2","Type":"ContainerDied","Data":"989ef8af1a3a07a07ebf4f20d9eb7d902b1cd405e4687bd031be9fec7ac9f372"}
Jul 17 05:37:40.338485 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.338319    2582 scope.go:117] "RemoveContainer" containerID="a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3"
Jul 17 05:37:40.346180 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.346164    2582 scope.go:117] "RemoveContainer" containerID="a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3"
Jul 17 05:37:40.346440 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:40.346425    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3\": container with ID starting with a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3 not found: ID does not exist" containerID="a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3"
Jul 17 05:37:40.346488 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.346455    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3"} err="failed to get container status \"a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3\": rpc error: code = NotFound desc = could not find container \"a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3\": container with ID starting with a6cfdf582b3d72c268a47babac013037c654473d4d7bd6a25d33c1f59450c9a3 not found: ID does not exist"
Jul 17 05:37:40.391962 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.391925    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-dshm\") pod \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") "
Jul 17 05:37:40.391962 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.391965    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tls-certs\") pod \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") "
Jul 17 05:37:40.392150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.391992    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-home\") pod \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") "
Jul 17 05:37:40.392150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.392025    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7829x\" (UniqueName: \"kubernetes.io/projected/10d39eba-1207-4069-a1b5-d7c0c6382fa2-kube-api-access-7829x\") pod \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") "
Jul 17 05:37:40.392150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.392058    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tmp-dir\") pod \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") "
Jul 17 05:37:40.392150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.392085    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-model-cache\") pod \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\" (UID: \"10d39eba-1207-4069-a1b5-d7c0c6382fa2\") "
Jul 17 05:37:40.392389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.392341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-home" (OuterVolumeSpecName: "home") pod "10d39eba-1207-4069-a1b5-d7c0c6382fa2" (UID: "10d39eba-1207-4069-a1b5-d7c0c6382fa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:37:40.392444 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.392404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10d39eba-1207-4069-a1b5-d7c0c6382fa2" (UID: "10d39eba-1207-4069-a1b5-d7c0c6382fa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:37:40.392483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.392429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-model-cache" (OuterVolumeSpecName: "model-cache") pod "10d39eba-1207-4069-a1b5-d7c0c6382fa2" (UID: "10d39eba-1207-4069-a1b5-d7c0c6382fa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:37:40.394368 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.394339    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-dshm" (OuterVolumeSpecName: "dshm") pod "10d39eba-1207-4069-a1b5-d7c0c6382fa2" (UID: "10d39eba-1207-4069-a1b5-d7c0c6382fa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:37:40.394468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.394426    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d39eba-1207-4069-a1b5-d7c0c6382fa2-kube-api-access-7829x" (OuterVolumeSpecName: "kube-api-access-7829x") pod "10d39eba-1207-4069-a1b5-d7c0c6382fa2" (UID: "10d39eba-1207-4069-a1b5-d7c0c6382fa2"). InnerVolumeSpecName "kube-api-access-7829x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:37:40.394468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.394429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10d39eba-1207-4069-a1b5-d7c0c6382fa2" (UID: "10d39eba-1207-4069-a1b5-d7c0c6382fa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:37:40.492997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.492945    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-dshm\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:40.492997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.492989    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tls-certs\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:40.492997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.492999    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-home\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:40.492997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.493008    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7829x\" (UniqueName: \"kubernetes.io/projected/10d39eba-1207-4069-a1b5-d7c0c6382fa2-kube-api-access-7829x\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:40.492997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.493017    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-tmp-dir\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:40.493326 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.493026    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d39eba-1207-4069-a1b5-d7c0c6382fa2-model-cache\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:40.659761 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.659732    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"]
Jul 17 05:37:40.664060 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.664035    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e1ab5d/test-model-e1ab5d-kserve-78c5f7bb49-wxbv4"]
Jul 17 05:37:40.730196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:40.730160    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d39eba-1207-4069-a1b5-d7c0c6382fa2" path="/var/lib/kubelet/pods/10d39eba-1207-4069-a1b5-d7c0c6382fa2/volumes"
Jul 17 05:37:45.724997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:45.724964    2582 scope.go:117] "RemoveContainer" containerID="6d9c723afee959e9a67b0dceb5b9b0bb376563aab7d9de6fd2d8fdb5065b640e"
Jul 17 05:37:46.368883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:46.368853    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerID="66ac1165fba9dd6c4e4b6bb588ecfc8d72b59869303b3dab547a4609bac5925f" exitCode=6
Jul 17 05:37:46.369050 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:46.368927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" event={"ID":"ffbf3017-b08b-469c-a21a-6993e9f5b93d","Type":"ContainerDied","Data":"66ac1165fba9dd6c4e4b6bb588ecfc8d72b59869303b3dab547a4609bac5925f"}
Jul 17 05:37:46.369050 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:46.368986    2582 scope.go:117] "RemoveContainer" containerID="6d9c723afee959e9a67b0dceb5b9b0bb376563aab7d9de6fd2d8fdb5065b640e"
Jul 17 05:37:46.369293 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:46.369275    2582 scope.go:117] "RemoveContainer" containerID="66ac1165fba9dd6c4e4b6bb588ecfc8d72b59869303b3dab547a4609bac5925f"
Jul 17 05:37:46.369506 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:46.369484    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj_odh-ai-gateway-infra(ffbf3017-b08b-469c-a21a-6993e9f5b93d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d"
Jul 17 05:37:47.396425 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:47.396342    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"]
Jul 17 05:37:47.532948 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:47.532922    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:47.654508 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:47.654396    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf3017-b08b-469c-a21a-6993e9f5b93d-maas-api-service-ca\") pod \"ffbf3017-b08b-469c-a21a-6993e9f5b93d\" (UID: \"ffbf3017-b08b-469c-a21a-6993e9f5b93d\") "
Jul 17 05:37:47.654799 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:47.654778    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbf3017-b08b-469c-a21a-6993e9f5b93d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffbf3017-b08b-469c-a21a-6993e9f5b93d" (UID: "ffbf3017-b08b-469c-a21a-6993e9f5b93d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:37:47.755286 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:47.755217    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf3017-b08b-469c-a21a-6993e9f5b93d-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:48.381402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:48.381373    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"
Jul 17 05:37:48.381598 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:48.381379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj" event={"ID":"ffbf3017-b08b-469c-a21a-6993e9f5b93d","Type":"ContainerDied","Data":"7bf5d0bb4c8938e2dc4d3f0b2a416b07de7bef62712af44ca6d7e218f55710d4"}
Jul 17 05:37:48.381598 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:48.381510    2582 scope.go:117] "RemoveContainer" containerID="66ac1165fba9dd6c4e4b6bb588ecfc8d72b59869303b3dab547a4609bac5925f"
Jul 17 05:37:48.407627 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:48.407388    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"]
Jul 17 05:37:48.409198 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:48.409174    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-f76zj"]
Jul 17 05:37:48.730314 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:48.730252    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" path="/var/lib/kubelet/pods/ffbf3017-b08b-469c-a21a-6993e9f5b93d/volumes"
Jul 17 05:37:49.725139 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:49.725108    2582 scope.go:117] "RemoveContainer" containerID="34afbee304ad6f79f4f9c40e7969e63882445bf9c727311c5f94767877dcd27b"
Jul 17 05:37:50.392715 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:50.392684    2582 generic.go:356] "Generic (PLEG): container finished" podID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerID="bf14827ad0853e49bf455917404c9b9f56ed9942edd3286aa857ea77cd7c492c" exitCode=6
Jul 17 05:37:50.392883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:50.392752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" event={"ID":"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8","Type":"ContainerDied","Data":"bf14827ad0853e49bf455917404c9b9f56ed9942edd3286aa857ea77cd7c492c"}
Jul 17 05:37:50.392883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:50.392809    2582 scope.go:117] "RemoveContainer" containerID="34afbee304ad6f79f4f9c40e7969e63882445bf9c727311c5f94767877dcd27b"
Jul 17 05:37:50.393149 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:50.393129    2582 scope.go:117] "RemoveContainer" containerID="bf14827ad0853e49bf455917404c9b9f56ed9942edd3286aa857ea77cd7c492c"
Jul 17 05:37:50.393375 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:50.393349    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z_odh-ai-gateway-infra(34c4d3b9-2d3c-49c2-8d09-596ef0761dc8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8"
Jul 17 05:37:51.129297 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129250    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"]
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129562    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129572    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129581    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129587    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d39eba-1207-4069-a1b5-d7c0c6382fa2" containerName="main"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129608    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d39eba-1207-4069-a1b5-d7c0c6382fa2" containerName="main"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129655    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129662    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="10d39eba-1207-4069-a1b5-d7c0c6382fa2" containerName="main"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129669    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.129680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.129675    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:51.133632 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.133614    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:37:51.140840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.140817    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"]
Jul 17 05:37:51.287448 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.287412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4763ad5-0048-47c2-b522-b279a44a68dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6rxbb\" (UID: \"b4763ad5-0048-47c2-b522-b279a44a68dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:37:51.388932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.388837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4763ad5-0048-47c2-b522-b279a44a68dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6rxbb\" (UID: \"b4763ad5-0048-47c2-b522-b279a44a68dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:37:51.389498 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.389477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4763ad5-0048-47c2-b522-b279a44a68dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-6rxbb\" (UID: \"b4763ad5-0048-47c2-b522-b279a44a68dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:37:51.418871 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.418840    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"]
Jul 17 05:37:51.444523 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.444492    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:37:51.577505 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.577477    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:51.595683 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.595659    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"]
Jul 17 05:37:51.598211 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:37:51.598177    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4763ad5_0048_47c2_b522_b279a44a68dd.slice/crio-82807b1b581943a7e8c55352068f2642a019618b8fa3176fed22c91302461f57 WatchSource:0}: Error finding container 82807b1b581943a7e8c55352068f2642a019618b8fa3176fed22c91302461f57: Status 404 returned error can't find the container with id 82807b1b581943a7e8c55352068f2642a019618b8fa3176fed22c91302461f57
Jul 17 05:37:51.690875 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.690851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8-maas-api-service-ca\") pod \"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8\" (UID: \"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8\") "
Jul 17 05:37:51.691169 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.691149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" (UID: "34c4d3b9-2d3c-49c2-8d09-596ef0761dc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:37:51.791968 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:51.791937    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:52.247819 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.247782    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"]
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248058    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248069    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248084    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248089    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248096    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248102    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248113    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248118    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbf3017-b08b-469c-a21a-6993e9f5b93d" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248168    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.248178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.248175    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:37:52.250961 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.250944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:52.259082 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.258999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"]
Jul 17 05:37:52.397369 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.397340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4fd38e-9dbd-44d8-b73e-87505e089eab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5\" (UID: \"cf4fd38e-9dbd-44d8-b73e-87505e089eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:52.404303 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.404276    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"
Jul 17 05:37:52.404477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.404296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z" event={"ID":"34c4d3b9-2d3c-49c2-8d09-596ef0761dc8","Type":"ContainerDied","Data":"baee68abd53bcdbce8f4b3d9f3bd8e1d1f10ad7a5b5e0fcff17ff2173ed14413"}
Jul 17 05:37:52.404477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.404366    2582 scope.go:117] "RemoveContainer" containerID="bf14827ad0853e49bf455917404c9b9f56ed9942edd3286aa857ea77cd7c492c"
Jul 17 05:37:52.405926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.405905    2582 generic.go:356] "Generic (PLEG): container finished" podID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerID="a67e76fac3a5aa532c9a4542950976a35d19966db4cb9d92209fcf46f0f8346f" exitCode=6
Jul 17 05:37:52.406065 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.405993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" event={"ID":"b4763ad5-0048-47c2-b522-b279a44a68dd","Type":"ContainerDied","Data":"a67e76fac3a5aa532c9a4542950976a35d19966db4cb9d92209fcf46f0f8346f"}
Jul 17 05:37:52.406065 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.406057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" event={"ID":"b4763ad5-0048-47c2-b522-b279a44a68dd","Type":"ContainerStarted","Data":"82807b1b581943a7e8c55352068f2642a019618b8fa3176fed22c91302461f57"}
Jul 17 05:37:52.406179 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.406131    2582 scope.go:117] "RemoveContainer" containerID="a67e76fac3a5aa532c9a4542950976a35d19966db4cb9d92209fcf46f0f8346f"
Jul 17 05:37:52.448463 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.448434    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"]
Jul 17 05:37:52.452912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.452890    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-m2r7z"]
Jul 17 05:37:52.498416 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.498389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4fd38e-9dbd-44d8-b73e-87505e089eab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5\" (UID: \"cf4fd38e-9dbd-44d8-b73e-87505e089eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:52.498910 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.498891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4fd38e-9dbd-44d8-b73e-87505e089eab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5\" (UID: \"cf4fd38e-9dbd-44d8-b73e-87505e089eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:52.561433 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.561380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:52.731499 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.731468    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" path="/var/lib/kubelet/pods/34c4d3b9-2d3c-49c2-8d09-596ef0761dc8/volumes"
Jul 17 05:37:52.777018 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:52.776995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"]
Jul 17 05:37:52.778722 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:37:52.778695    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4fd38e_9dbd_44d8_b73e_87505e089eab.slice/crio-1745aa098188439649b42704703c7a1ef73303df6c672137e31b777f2bf6f80b WatchSource:0}: Error finding container 1745aa098188439649b42704703c7a1ef73303df6c672137e31b777f2bf6f80b: Status 404 returned error can't find the container with id 1745aa098188439649b42704703c7a1ef73303df6c672137e31b777f2bf6f80b
Jul 17 05:37:53.412957 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.412851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5" event={"ID":"cf4fd38e-9dbd-44d8-b73e-87505e089eab","Type":"ContainerStarted","Data":"b079cf15a3922dd747a64921c0ec9f6d5e8036b49e07360e90615050ba6ab9fa"}
Jul 17 05:37:53.412957 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.412943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5" event={"ID":"cf4fd38e-9dbd-44d8-b73e-87505e089eab","Type":"ContainerStarted","Data":"1745aa098188439649b42704703c7a1ef73303df6c672137e31b777f2bf6f80b"}
Jul 17 05:37:53.414879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.414849    2582 generic.go:356] "Generic (PLEG): container finished" podID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerID="1b86a45530152876716731bc81647edd219c435199c73672298dac7650269d71" exitCode=6
Jul 17 05:37:53.415062 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.414911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" event={"ID":"b4763ad5-0048-47c2-b522-b279a44a68dd","Type":"ContainerDied","Data":"1b86a45530152876716731bc81647edd219c435199c73672298dac7650269d71"}
Jul 17 05:37:53.415170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.415108    2582 scope.go:117] "RemoveContainer" containerID="a67e76fac3a5aa532c9a4542950976a35d19966db4cb9d92209fcf46f0f8346f"
Jul 17 05:37:53.415219 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.415175    2582 scope.go:117] "RemoveContainer" containerID="1b86a45530152876716731bc81647edd219c435199c73672298dac7650269d71"
Jul 17 05:37:53.415414 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:53.415396    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6rxbb_odh-ai-gateway-infra(b4763ad5-0048-47c2-b522-b279a44a68dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd"
Jul 17 05:37:53.430353 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:53.430308    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5" podStartSLOduration=1.43029227 podStartE2EDuration="1.43029227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:37:53.428721031 +0000 UTC m=+2877.165783448" watchObservedRunningTime="2026-07-17 05:37:53.43029227 +0000 UTC m=+2877.167354629"
Jul 17 05:37:54.421172 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:54.421131    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf4fd38e-9dbd-44d8-b73e-87505e089eab" containerID="b079cf15a3922dd747a64921c0ec9f6d5e8036b49e07360e90615050ba6ab9fa" exitCode=0
Jul 17 05:37:54.421683 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:54.421179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5" event={"ID":"cf4fd38e-9dbd-44d8-b73e-87505e089eab","Type":"ContainerDied","Data":"b079cf15a3922dd747a64921c0ec9f6d5e8036b49e07360e90615050ba6ab9fa"}
Jul 17 05:37:54.423085 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:54.423067    2582 scope.go:117] "RemoveContainer" containerID="1b86a45530152876716731bc81647edd219c435199c73672298dac7650269d71"
Jul 17 05:37:54.423330 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:37:54.423309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6rxbb_odh-ai-gateway-infra(b4763ad5-0048-47c2-b522-b279a44a68dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd"
Jul 17 05:37:55.559779 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:55.559751    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:55.725236 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:55.725198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4fd38e-9dbd-44d8-b73e-87505e089eab-maas-api-service-ca\") pod \"cf4fd38e-9dbd-44d8-b73e-87505e089eab\" (UID: \"cf4fd38e-9dbd-44d8-b73e-87505e089eab\") "
Jul 17 05:37:55.725595 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:55.725573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4fd38e-9dbd-44d8-b73e-87505e089eab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf4fd38e-9dbd-44d8-b73e-87505e089eab" (UID: "cf4fd38e-9dbd-44d8-b73e-87505e089eab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:37:55.826597 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:55.826558    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4fd38e-9dbd-44d8-b73e-87505e089eab-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:37:56.438155 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:56.438119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5" event={"ID":"cf4fd38e-9dbd-44d8-b73e-87505e089eab","Type":"ContainerDied","Data":"1745aa098188439649b42704703c7a1ef73303df6c672137e31b777f2bf6f80b"}
Jul 17 05:37:56.438155 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:56.438150    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5"
Jul 17 05:37:56.438418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:37:56.438209    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1745aa098188439649b42704703c7a1ef73303df6c672137e31b777f2bf6f80b"
Jul 17 05:38:06.726974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:06.726943    2582 scope.go:117] "RemoveContainer" containerID="1b86a45530152876716731bc81647edd219c435199c73672298dac7650269d71"
Jul 17 05:38:07.354060 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.354028    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"]
Jul 17 05:38:07.354376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.354363    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4fd38e-9dbd-44d8-b73e-87505e089eab" containerName="revoke-keys"
Jul 17 05:38:07.354425 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.354379    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4fd38e-9dbd-44d8-b73e-87505e089eab" containerName="revoke-keys"
Jul 17 05:38:07.354466 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.354455    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c4d3b9-2d3c-49c2-8d09-596ef0761dc8" containerName="revoke-keys"
Jul 17 05:38:07.354500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.354471    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4fd38e-9dbd-44d8-b73e-87505e089eab" containerName="revoke-keys"
Jul 17 05:38:07.357206 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.357184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:07.365740 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.365704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"]
Jul 17 05:38:07.415730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.415695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5470df73-66a5-4b2e-8728-f18eed113175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8\" (UID: \"5470df73-66a5-4b2e-8728-f18eed113175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:07.485767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.485735    2582 generic.go:356] "Generic (PLEG): container finished" podID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerID="06667e89aa7fdb17a5cb993c9b17bd465cb5d0e7bba396d0ff674e1554db856a" exitCode=6
Jul 17 05:38:07.485928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.485796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" event={"ID":"b4763ad5-0048-47c2-b522-b279a44a68dd","Type":"ContainerDied","Data":"06667e89aa7fdb17a5cb993c9b17bd465cb5d0e7bba396d0ff674e1554db856a"}
Jul 17 05:38:07.485928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.485875    2582 scope.go:117] "RemoveContainer" containerID="1b86a45530152876716731bc81647edd219c435199c73672298dac7650269d71"
Jul 17 05:38:07.486283 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.486243    2582 scope.go:117] "RemoveContainer" containerID="06667e89aa7fdb17a5cb993c9b17bd465cb5d0e7bba396d0ff674e1554db856a"
Jul 17 05:38:07.486490 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:07.486464    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-6rxbb_odh-ai-gateway-infra(b4763ad5-0048-47c2-b522-b279a44a68dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd"
Jul 17 05:38:07.516910 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.516881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5470df73-66a5-4b2e-8728-f18eed113175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8\" (UID: \"5470df73-66a5-4b2e-8728-f18eed113175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:07.517462 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.517445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5470df73-66a5-4b2e-8728-f18eed113175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8\" (UID: \"5470df73-66a5-4b2e-8728-f18eed113175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:07.667511 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.667420    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:07.801115 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:07.801092    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"]
Jul 17 05:38:07.803468 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:07.803444    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5470df73_66a5_4b2e_8728_f18eed113175.slice/crio-3614aed0891ee261b3e45004be983667727734c721cd9973d649a390fb712847 WatchSource:0}: Error finding container 3614aed0891ee261b3e45004be983667727734c721cd9973d649a390fb712847: Status 404 returned error can't find the container with id 3614aed0891ee261b3e45004be983667727734c721cd9973d649a390fb712847
Jul 17 05:38:08.491813 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.491770    2582 generic.go:356] "Generic (PLEG): container finished" podID="5470df73-66a5-4b2e-8728-f18eed113175" containerID="f5ace8f1e40c69b5c74180e4fec9e77304e5c98bb20a70711cae377805f139b7" exitCode=6
Jul 17 05:38:08.491987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.491858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" event={"ID":"5470df73-66a5-4b2e-8728-f18eed113175","Type":"ContainerDied","Data":"f5ace8f1e40c69b5c74180e4fec9e77304e5c98bb20a70711cae377805f139b7"}
Jul 17 05:38:08.491987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.491920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" event={"ID":"5470df73-66a5-4b2e-8728-f18eed113175","Type":"ContainerStarted","Data":"3614aed0891ee261b3e45004be983667727734c721cd9973d649a390fb712847"}
Jul 17 05:38:08.492106 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.492059    2582 scope.go:117] "RemoveContainer" containerID="f5ace8f1e40c69b5c74180e4fec9e77304e5c98bb20a70711cae377805f139b7"
Jul 17 05:38:08.524365 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.524337    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"]
Jul 17 05:38:08.652887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.652865    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:38:08.727205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.727177    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4763ad5-0048-47c2-b522-b279a44a68dd-maas-api-service-ca\") pod \"b4763ad5-0048-47c2-b522-b279a44a68dd\" (UID: \"b4763ad5-0048-47c2-b522-b279a44a68dd\") "
Jul 17 05:38:08.727554 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.727531    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4763ad5-0048-47c2-b522-b279a44a68dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4763ad5-0048-47c2-b522-b279a44a68dd" (UID: "b4763ad5-0048-47c2-b522-b279a44a68dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:38:08.828398 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:08.828368    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4763ad5-0048-47c2-b522-b279a44a68dd-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:38:09.499786 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.499756    2582 generic.go:356] "Generic (PLEG): container finished" podID="5470df73-66a5-4b2e-8728-f18eed113175" containerID="f947949c9280315ec4ac1a71c9262501fb14f98af3a0f9ce0cc42f47db1b2c41" exitCode=6
Jul 17 05:38:09.499978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.499838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" event={"ID":"5470df73-66a5-4b2e-8728-f18eed113175","Type":"ContainerDied","Data":"f947949c9280315ec4ac1a71c9262501fb14f98af3a0f9ce0cc42f47db1b2c41"}
Jul 17 05:38:09.499978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.499906    2582 scope.go:117] "RemoveContainer" containerID="f5ace8f1e40c69b5c74180e4fec9e77304e5c98bb20a70711cae377805f139b7"
Jul 17 05:38:09.500132 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.500115    2582 scope.go:117] "RemoveContainer" containerID="f947949c9280315ec4ac1a71c9262501fb14f98af3a0f9ce0cc42f47db1b2c41"
Jul 17 05:38:09.500366 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:09.500346    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8_odh-ai-gateway-infra(5470df73-66a5-4b2e-8728-f18eed113175)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" podUID="5470df73-66a5-4b2e-8728-f18eed113175"
Jul 17 05:38:09.501146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.501016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb" event={"ID":"b4763ad5-0048-47c2-b522-b279a44a68dd","Type":"ContainerDied","Data":"82807b1b581943a7e8c55352068f2642a019618b8fa3176fed22c91302461f57"}
Jul 17 05:38:09.501146 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.501042    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"
Jul 17 05:38:09.510358 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.510337    2582 scope.go:117] "RemoveContainer" containerID="06667e89aa7fdb17a5cb993c9b17bd465cb5d0e7bba396d0ff674e1554db856a"
Jul 17 05:38:09.531926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.531899    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"]
Jul 17 05:38:09.536307 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:09.536283    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-6rxbb"]
Jul 17 05:38:10.507926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:10.507896    2582 scope.go:117] "RemoveContainer" containerID="f947949c9280315ec4ac1a71c9262501fb14f98af3a0f9ce0cc42f47db1b2c41"
Jul 17 05:38:10.508415 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:10.508138    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8_odh-ai-gateway-infra(5470df73-66a5-4b2e-8728-f18eed113175)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" podUID="5470df73-66a5-4b2e-8728-f18eed113175"
Jul 17 05:38:10.729509 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:10.729476    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" path="/var/lib/kubelet/pods/b4763ad5-0048-47c2-b522-b279a44a68dd/volumes"
Jul 17 05:38:19.294199 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294162    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"]
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294466    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294478    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294494    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294499    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294510    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294515    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294564    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294573    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.294603 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.294580    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4763ad5-0048-47c2-b522-b279a44a68dd" containerName="revoke-keys"
Jul 17 05:38:19.299025 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.299002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:19.307528 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.307500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"]
Jul 17 05:38:19.416662 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.416623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbcd7f-5f05-4347-8c2b-279bc267116f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h\" (UID: \"f0cbcd7f-5f05-4347-8c2b-279bc267116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:19.517846 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.517814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbcd7f-5f05-4347-8c2b-279bc267116f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h\" (UID: \"f0cbcd7f-5f05-4347-8c2b-279bc267116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:19.518468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.518446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbcd7f-5f05-4347-8c2b-279bc267116f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h\" (UID: \"f0cbcd7f-5f05-4347-8c2b-279bc267116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:19.610800 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.610695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:19.745034 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:19.745007    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"]
Jul 17 05:38:19.747579 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:19.747553    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cbcd7f_5f05_4347_8c2b_279bc267116f.slice/crio-1df8994a7d99ab8c71f499cf5426a6167f1b10abf55aadad3e11f75b98622655 WatchSource:0}: Error finding container 1df8994a7d99ab8c71f499cf5426a6167f1b10abf55aadad3e11f75b98622655: Status 404 returned error can't find the container with id 1df8994a7d99ab8c71f499cf5426a6167f1b10abf55aadad3e11f75b98622655
Jul 17 05:38:20.557706 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:20.557674    2582 generic.go:356] "Generic (PLEG): container finished" podID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerID="c3f7829b6261fac20bd47263c8656d708e9f38baff00d382ad307ca9f9bfd4e5" exitCode=6
Jul 17 05:38:20.558112 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:20.557726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" event={"ID":"f0cbcd7f-5f05-4347-8c2b-279bc267116f","Type":"ContainerDied","Data":"c3f7829b6261fac20bd47263c8656d708e9f38baff00d382ad307ca9f9bfd4e5"}
Jul 17 05:38:20.558112 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:20.557764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" event={"ID":"f0cbcd7f-5f05-4347-8c2b-279bc267116f","Type":"ContainerStarted","Data":"1df8994a7d99ab8c71f499cf5426a6167f1b10abf55aadad3e11f75b98622655"}
Jul 17 05:38:20.558112 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:20.557965    2582 scope.go:117] "RemoveContainer" containerID="c3f7829b6261fac20bd47263c8656d708e9f38baff00d382ad307ca9f9bfd4e5"
Jul 17 05:38:21.564009 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:21.563912    2582 generic.go:356] "Generic (PLEG): container finished" podID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerID="c7b219dc23282b19aa8f448e8b7c95b9c0be51153f8f167f953791b6327f2c7f" exitCode=6
Jul 17 05:38:21.564009 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:21.563982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" event={"ID":"f0cbcd7f-5f05-4347-8c2b-279bc267116f","Type":"ContainerDied","Data":"c7b219dc23282b19aa8f448e8b7c95b9c0be51153f8f167f953791b6327f2c7f"}
Jul 17 05:38:21.564578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:21.564035    2582 scope.go:117] "RemoveContainer" containerID="c3f7829b6261fac20bd47263c8656d708e9f38baff00d382ad307ca9f9bfd4e5"
Jul 17 05:38:21.564578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:21.564246    2582 scope.go:117] "RemoveContainer" containerID="c7b219dc23282b19aa8f448e8b7c95b9c0be51153f8f167f953791b6327f2c7f"
Jul 17 05:38:21.564578 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:21.564500    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h_odh-ai-gateway-infra(f0cbcd7f-5f05-4347-8c2b-279bc267116f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f"
Jul 17 05:38:22.569921 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:22.569890    2582 scope.go:117] "RemoveContainer" containerID="c7b219dc23282b19aa8f448e8b7c95b9c0be51153f8f167f953791b6327f2c7f"
Jul 17 05:38:22.570330 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:22.570053    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h_odh-ai-gateway-infra(f0cbcd7f-5f05-4347-8c2b-279bc267116f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f"
Jul 17 05:38:22.724889 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:22.724857    2582 scope.go:117] "RemoveContainer" containerID="f947949c9280315ec4ac1a71c9262501fb14f98af3a0f9ce0cc42f47db1b2c41"
Jul 17 05:38:23.575515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:23.575423    2582 generic.go:356] "Generic (PLEG): container finished" podID="5470df73-66a5-4b2e-8728-f18eed113175" containerID="7b449c1c75bfe8b460606ac01de2a0b7255dea5cc53894196ae22168597c3fe1" exitCode=6
Jul 17 05:38:23.575515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:23.575470    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" event={"ID":"5470df73-66a5-4b2e-8728-f18eed113175","Type":"ContainerDied","Data":"7b449c1c75bfe8b460606ac01de2a0b7255dea5cc53894196ae22168597c3fe1"}
Jul 17 05:38:23.575983 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:23.575533    2582 scope.go:117] "RemoveContainer" containerID="f947949c9280315ec4ac1a71c9262501fb14f98af3a0f9ce0cc42f47db1b2c41"
Jul 17 05:38:23.575983 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:23.575880    2582 scope.go:117] "RemoveContainer" containerID="7b449c1c75bfe8b460606ac01de2a0b7255dea5cc53894196ae22168597c3fe1"
Jul 17 05:38:23.576114 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:23.576094    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8_odh-ai-gateway-infra(5470df73-66a5-4b2e-8728-f18eed113175)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" podUID="5470df73-66a5-4b2e-8728-f18eed113175"
Jul 17 05:38:24.619558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.619522    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"]
Jul 17 05:38:24.724153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.724118    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"]
Jul 17 05:38:24.729233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.729055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:24.735848 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.735789    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"]
Jul 17 05:38:24.760362 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.760341    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:24.864098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.864063    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5470df73-66a5-4b2e-8728-f18eed113175-maas-api-service-ca\") pod \"5470df73-66a5-4b2e-8728-f18eed113175\" (UID: \"5470df73-66a5-4b2e-8728-f18eed113175\") "
Jul 17 05:38:24.864305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.864248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1be9f6-8673-4430-8f63-3af7c508dec6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd\" (UID: \"1c1be9f6-8673-4430-8f63-3af7c508dec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:24.864447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.864427    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5470df73-66a5-4b2e-8728-f18eed113175-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5470df73-66a5-4b2e-8728-f18eed113175" (UID: "5470df73-66a5-4b2e-8728-f18eed113175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:38:24.965122 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.965095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1be9f6-8673-4430-8f63-3af7c508dec6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd\" (UID: \"1c1be9f6-8673-4430-8f63-3af7c508dec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:24.965322 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.965167    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5470df73-66a5-4b2e-8728-f18eed113175-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:38:24.965733 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:24.965714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1be9f6-8673-4430-8f63-3af7c508dec6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd\" (UID: \"1c1be9f6-8673-4430-8f63-3af7c508dec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:25.056824 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.056780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:25.189940 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.189914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"]
Jul 17 05:38:25.192069 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:25.192041    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1be9f6_8673_4430_8f63_3af7c508dec6.slice/crio-36d364d477ad4e7f00c683c39daee5d470352abeba8fb81b7e70829bf3f2f77e WatchSource:0}: Error finding container 36d364d477ad4e7f00c683c39daee5d470352abeba8fb81b7e70829bf3f2f77e: Status 404 returned error can't find the container with id 36d364d477ad4e7f00c683c39daee5d470352abeba8fb81b7e70829bf3f2f77e
Jul 17 05:38:25.591984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.591899    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"
Jul 17 05:38:25.591984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.591932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8" event={"ID":"5470df73-66a5-4b2e-8728-f18eed113175","Type":"ContainerDied","Data":"3614aed0891ee261b3e45004be983667727734c721cd9973d649a390fb712847"}
Jul 17 05:38:25.592212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.591986    2582 scope.go:117] "RemoveContainer" containerID="7b449c1c75bfe8b460606ac01de2a0b7255dea5cc53894196ae22168597c3fe1"
Jul 17 05:38:25.593587 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.593562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" event={"ID":"1c1be9f6-8673-4430-8f63-3af7c508dec6","Type":"ContainerStarted","Data":"1a2d153852a5c4684e18e90f2328e737c4420c6b0b69af80a80125c8f5184a49"}
Jul 17 05:38:25.593698 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.593614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" event={"ID":"1c1be9f6-8673-4430-8f63-3af7c508dec6","Type":"ContainerStarted","Data":"36d364d477ad4e7f00c683c39daee5d470352abeba8fb81b7e70829bf3f2f77e"}
Jul 17 05:38:25.611929 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.611889    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" podStartSLOduration=1.611875724 podStartE2EDuration="1.611875724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:38:25.610837947 +0000 UTC m=+2909.347900282" watchObservedRunningTime="2026-07-17 05:38:25.611875724 +0000 UTC m=+2909.348938059"
Jul 17 05:38:25.623010 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.622985    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"]
Jul 17 05:38:25.625541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:25.625515    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-crbd8"]
Jul 17 05:38:26.599748 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:26.599717    2582 generic.go:356] "Generic (PLEG): container finished" podID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerID="1a2d153852a5c4684e18e90f2328e737c4420c6b0b69af80a80125c8f5184a49" exitCode=6
Jul 17 05:38:26.600035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:26.599751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" event={"ID":"1c1be9f6-8673-4430-8f63-3af7c508dec6","Type":"ContainerDied","Data":"1a2d153852a5c4684e18e90f2328e737c4420c6b0b69af80a80125c8f5184a49"}
Jul 17 05:38:26.600035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:26.600018    2582 scope.go:117] "RemoveContainer" containerID="1a2d153852a5c4684e18e90f2328e737c4420c6b0b69af80a80125c8f5184a49"
Jul 17 05:38:26.729633 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:26.729600    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5470df73-66a5-4b2e-8728-f18eed113175" path="/var/lib/kubelet/pods/5470df73-66a5-4b2e-8728-f18eed113175/volumes"
Jul 17 05:38:27.607057 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:27.607028    2582 generic.go:356] "Generic (PLEG): container finished" podID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerID="68825b0a9e6f3630ffe7a9746e2beff7857f03e3c6f5a4f705d874c40923e2b2" exitCode=6
Jul 17 05:38:27.607271 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:27.607084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" event={"ID":"1c1be9f6-8673-4430-8f63-3af7c508dec6","Type":"ContainerDied","Data":"68825b0a9e6f3630ffe7a9746e2beff7857f03e3c6f5a4f705d874c40923e2b2"}
Jul 17 05:38:27.607271 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:27.607130    2582 scope.go:117] "RemoveContainer" containerID="1a2d153852a5c4684e18e90f2328e737c4420c6b0b69af80a80125c8f5184a49"
Jul 17 05:38:27.607404 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:27.607339    2582 scope.go:117] "RemoveContainer" containerID="68825b0a9e6f3630ffe7a9746e2beff7857f03e3c6f5a4f705d874c40923e2b2"
Jul 17 05:38:27.607589 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:27.607568    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd_odh-ai-gateway-infra(1c1be9f6-8673-4430-8f63-3af7c508dec6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6"
Jul 17 05:38:28.612490 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:28.612467    2582 scope.go:117] "RemoveContainer" containerID="68825b0a9e6f3630ffe7a9746e2beff7857f03e3c6f5a4f705d874c40923e2b2"
Jul 17 05:38:28.612872 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:28.612643    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd_odh-ai-gateway-infra(1c1be9f6-8673-4430-8f63-3af7c508dec6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6"
Jul 17 05:38:30.783661 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.783626    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"]
Jul 17 05:38:30.784023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.783910    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:30.784023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.783921    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:30.784023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.783937    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:30.784023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.783942    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:30.784023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.783992    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:30.784023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.784001    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:30.788208 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.788182    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:30.791615 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.791588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xwcd7\""
Jul 17 05:38:30.792075 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.792052    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ce2966-serving-cert\""
Jul 17 05:38:30.792711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.792691    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 05:38:30.799915 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.799850    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"]
Jul 17 05:38:30.920106 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.920068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svs8\" (UniqueName: \"kubernetes.io/projected/36114828-85bc-452c-afbc-b84e0763a16a-kube-api-access-7svs8\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:30.920333 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.920137    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36114828-85bc-452c-afbc-b84e0763a16a-openshift-service-ca\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:30.920333 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:30.920194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36114828-85bc-452c-afbc-b84e0763a16a-maas-api-tls\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.021192 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.021159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36114828-85bc-452c-afbc-b84e0763a16a-openshift-service-ca\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.021399 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.021207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36114828-85bc-452c-afbc-b84e0763a16a-maas-api-tls\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.021399 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.021244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svs8\" (UniqueName: \"kubernetes.io/projected/36114828-85bc-452c-afbc-b84e0763a16a-kube-api-access-7svs8\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.021951 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.021930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36114828-85bc-452c-afbc-b84e0763a16a-openshift-service-ca\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.023774 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.023757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36114828-85bc-452c-afbc-b84e0763a16a-maas-api-tls\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.032758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.032735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svs8\" (UniqueName: \"kubernetes.io/projected/36114828-85bc-452c-afbc-b84e0763a16a-kube-api-access-7svs8\") pod \"maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.102141 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.102040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:31.241074 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.241047    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"]
Jul 17 05:38:31.243317 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:31.243293    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36114828_85bc_452c_afbc_b84e0763a16a.slice/crio-e0f6e26a48efb34f3c4fb7f6afba84877c37904665109d3f826a6f8622ce9ddb WatchSource:0}: Error finding container e0f6e26a48efb34f3c4fb7f6afba84877c37904665109d3f826a6f8622ce9ddb: Status 404 returned error can't find the container with id e0f6e26a48efb34f3c4fb7f6afba84877c37904665109d3f826a6f8622ce9ddb
Jul 17 05:38:31.244798 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.244781    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 05:38:31.633359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:31.633320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68" event={"ID":"36114828-85bc-452c-afbc-b84e0763a16a","Type":"ContainerStarted","Data":"e0f6e26a48efb34f3c4fb7f6afba84877c37904665109d3f826a6f8622ce9ddb"}
Jul 17 05:38:33.645085 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:33.645051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68" event={"ID":"36114828-85bc-452c-afbc-b84e0763a16a","Type":"ContainerStarted","Data":"f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba"}
Jul 17 05:38:33.645475 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:33.645174    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:33.665423 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:33.665373    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68" podStartSLOduration=1.358409926 podStartE2EDuration="3.665354094s" totalImagesPullingTime="2.306944168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:38:33.663134533 +0000 UTC m=+2917.400196870" watchObservedRunningTime="2026-07-17 05:38:33.665354094 +0000 UTC m=+2917.402416430"
Jul 17 05:38:34.725018 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:34.724985    2582 scope.go:117] "RemoveContainer" containerID="c7b219dc23282b19aa8f448e8b7c95b9c0be51153f8f167f953791b6327f2c7f"
Jul 17 05:38:35.655986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:35.655895    2582 generic.go:356] "Generic (PLEG): container finished" podID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerID="62473b86a4dc39b8c625d4d89d7534037ec9d147c3a8c5b197e5c4fb23c1ea2a" exitCode=6
Jul 17 05:38:35.655986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:35.655964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" event={"ID":"f0cbcd7f-5f05-4347-8c2b-279bc267116f","Type":"ContainerDied","Data":"62473b86a4dc39b8c625d4d89d7534037ec9d147c3a8c5b197e5c4fb23c1ea2a"}
Jul 17 05:38:35.656277 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:35.656045    2582 scope.go:117] "RemoveContainer" containerID="c7b219dc23282b19aa8f448e8b7c95b9c0be51153f8f167f953791b6327f2c7f"
Jul 17 05:38:35.656474 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:35.656460    2582 scope.go:117] "RemoveContainer" containerID="62473b86a4dc39b8c625d4d89d7534037ec9d147c3a8c5b197e5c4fb23c1ea2a"
Jul 17 05:38:35.656731 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:35.656695    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h_odh-ai-gateway-infra(f0cbcd7f-5f05-4347-8c2b-279bc267116f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f"
Jul 17 05:38:36.686221 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:36.686176    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"]
Jul 17 05:38:36.816270 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:36.816237    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:36.990545 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:36.990511    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbcd7f-5f05-4347-8c2b-279bc267116f-maas-api-service-ca\") pod \"f0cbcd7f-5f05-4347-8c2b-279bc267116f\" (UID: \"f0cbcd7f-5f05-4347-8c2b-279bc267116f\") "
Jul 17 05:38:36.990937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:36.990886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0cbcd7f-5f05-4347-8c2b-279bc267116f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0cbcd7f-5f05-4347-8c2b-279bc267116f" (UID: "f0cbcd7f-5f05-4347-8c2b-279bc267116f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:38:37.091476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:37.091435    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbcd7f-5f05-4347-8c2b-279bc267116f-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:38:37.668500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:37.668467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h" event={"ID":"f0cbcd7f-5f05-4347-8c2b-279bc267116f","Type":"ContainerDied","Data":"1df8994a7d99ab8c71f499cf5426a6167f1b10abf55aadad3e11f75b98622655"}
Jul 17 05:38:37.668500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:37.668491    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"
Jul 17 05:38:37.668786 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:37.668532    2582 scope.go:117] "RemoveContainer" containerID="62473b86a4dc39b8c625d4d89d7534037ec9d147c3a8c5b197e5c4fb23c1ea2a"
Jul 17 05:38:37.690143 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:37.690113    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"]
Jul 17 05:38:37.700949 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:37.700918    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-m6p7h"]
Jul 17 05:38:38.729796 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:38.729768    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" path="/var/lib/kubelet/pods/f0cbcd7f-5f05-4347-8c2b-279bc267116f/volumes"
Jul 17 05:38:39.654617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:39.654591    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:38:41.175282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175229    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"]
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175561    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175571    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175582    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175587    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175594    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175610    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175615    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175659    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.175664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175664    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5470df73-66a5-4b2e-8728-f18eed113175" containerName="revoke-keys"
Jul 17 05:38:41.175977 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.175672    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:41.179906 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.179890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:38:41.185487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.185458    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"]
Jul 17 05:38:41.328781 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.328741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccf528c-0888-42f4-bdf8-50f6bb80f174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-2dxxt\" (UID: \"bccf528c-0888-42f4-bdf8-50f6bb80f174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:38:41.429490 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.429405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccf528c-0888-42f4-bdf8-50f6bb80f174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-2dxxt\" (UID: \"bccf528c-0888-42f4-bdf8-50f6bb80f174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:38:41.430012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.429995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccf528c-0888-42f4-bdf8-50f6bb80f174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-2dxxt\" (UID: \"bccf528c-0888-42f4-bdf8-50f6bb80f174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:38:41.492477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.492443    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:38:41.632688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.632447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"]
Jul 17 05:38:41.634929 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:41.634902    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccf528c_0888_42f4_bdf8_50f6bb80f174.slice/crio-4328a56024af83b7f56dbbd13019d3a5cdb896d601bfe40ae55244c284824224 WatchSource:0}: Error finding container 4328a56024af83b7f56dbbd13019d3a5cdb896d601bfe40ae55244c284824224: Status 404 returned error can't find the container with id 4328a56024af83b7f56dbbd13019d3a5cdb896d601bfe40ae55244c284824224
Jul 17 05:38:41.689832 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.689804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" event={"ID":"bccf528c-0888-42f4-bdf8-50f6bb80f174","Type":"ContainerStarted","Data":"4328a56024af83b7f56dbbd13019d3a5cdb896d601bfe40ae55244c284824224"}
Jul 17 05:38:41.725075 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:41.725048    2582 scope.go:117] "RemoveContainer" containerID="68825b0a9e6f3630ffe7a9746e2beff7857f03e3c6f5a4f705d874c40923e2b2"
Jul 17 05:38:42.695971 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.695942    2582 generic.go:356] "Generic (PLEG): container finished" podID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerID="42ed622ca893dd2041f05316788bdf4971a9a7e5ca92e2a8a9d6cd4e00ddaef4" exitCode=6
Jul 17 05:38:42.696446 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.696010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" event={"ID":"1c1be9f6-8673-4430-8f63-3af7c508dec6","Type":"ContainerDied","Data":"42ed622ca893dd2041f05316788bdf4971a9a7e5ca92e2a8a9d6cd4e00ddaef4"}
Jul 17 05:38:42.696446 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.696118    2582 scope.go:117] "RemoveContainer" containerID="68825b0a9e6f3630ffe7a9746e2beff7857f03e3c6f5a4f705d874c40923e2b2"
Jul 17 05:38:42.696575 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.696467    2582 scope.go:117] "RemoveContainer" containerID="42ed622ca893dd2041f05316788bdf4971a9a7e5ca92e2a8a9d6cd4e00ddaef4"
Jul 17 05:38:42.696717 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:42.696692    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd_odh-ai-gateway-infra(1c1be9f6-8673-4430-8f63-3af7c508dec6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6"
Jul 17 05:38:42.697619 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.697599    2582 generic.go:356] "Generic (PLEG): container finished" podID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerID="550c8689278e790d1bf54526e374e012e5a61ad5f671e3890d638e1a83bedde0" exitCode=6
Jul 17 05:38:42.697703 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.697648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" event={"ID":"bccf528c-0888-42f4-bdf8-50f6bb80f174","Type":"ContainerDied","Data":"550c8689278e790d1bf54526e374e012e5a61ad5f671e3890d638e1a83bedde0"}
Jul 17 05:38:42.697820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:42.697808    2582 scope.go:117] "RemoveContainer" containerID="550c8689278e790d1bf54526e374e012e5a61ad5f671e3890d638e1a83bedde0"
Jul 17 05:38:43.262143 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.262107    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"]
Jul 17 05:38:43.262557 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.262539    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0cbcd7f-5f05-4347-8c2b-279bc267116f" containerName="revoke-keys"
Jul 17 05:38:43.265447 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.265427    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:43.272620 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.272594    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"]
Jul 17 05:38:43.446969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.446926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d6fe0b-727d-42da-8cbb-939697dbdf82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj\" (UID: \"57d6fe0b-727d-42da-8cbb-939697dbdf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:43.547806 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.547702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d6fe0b-727d-42da-8cbb-939697dbdf82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj\" (UID: \"57d6fe0b-727d-42da-8cbb-939697dbdf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:43.548363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.548344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d6fe0b-727d-42da-8cbb-939697dbdf82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj\" (UID: \"57d6fe0b-727d-42da-8cbb-939697dbdf82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:43.576933 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.576906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:43.704133 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.704105    2582 generic.go:356] "Generic (PLEG): container finished" podID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerID="890017505d96c230937ab81f9cc5004cdf69e1d8ff20f5d832d9e66cd8904445" exitCode=6
Jul 17 05:38:43.704613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.704272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" event={"ID":"bccf528c-0888-42f4-bdf8-50f6bb80f174","Type":"ContainerDied","Data":"890017505d96c230937ab81f9cc5004cdf69e1d8ff20f5d832d9e66cd8904445"}
Jul 17 05:38:43.704613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.704348    2582 scope.go:117] "RemoveContainer" containerID="550c8689278e790d1bf54526e374e012e5a61ad5f671e3890d638e1a83bedde0"
Jul 17 05:38:43.704613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.704585    2582 scope.go:117] "RemoveContainer" containerID="890017505d96c230937ab81f9cc5004cdf69e1d8ff20f5d832d9e66cd8904445"
Jul 17 05:38:43.704825 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:43.704802    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-2dxxt_odh-ai-gateway-infra(bccf528c-0888-42f4-bdf8-50f6bb80f174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174"
Jul 17 05:38:43.707399 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.707376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"]
Jul 17 05:38:43.709772 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:43.709744    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d6fe0b_727d_42da_8cbb_939697dbdf82.slice/crio-d0bbcccaf0453c917d82b9af1db99846d40743b9f4a20c347cbe50fa53575239 WatchSource:0}: Error finding container d0bbcccaf0453c917d82b9af1db99846d40743b9f4a20c347cbe50fa53575239: Status 404 returned error can't find the container with id d0bbcccaf0453c917d82b9af1db99846d40743b9f4a20c347cbe50fa53575239
Jul 17 05:38:43.722813 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.722743    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"]
Jul 17 05:38:43.855675 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.855652    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:43.950040 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.950007    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1be9f6-8673-4430-8f63-3af7c508dec6-maas-api-service-ca\") pod \"1c1be9f6-8673-4430-8f63-3af7c508dec6\" (UID: \"1c1be9f6-8673-4430-8f63-3af7c508dec6\") "
Jul 17 05:38:43.950433 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:43.950398    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1be9f6-8673-4430-8f63-3af7c508dec6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c1be9f6-8673-4430-8f63-3af7c508dec6" (UID: "1c1be9f6-8673-4430-8f63-3af7c508dec6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:38:44.051535 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.051491    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1be9f6-8673-4430-8f63-3af7c508dec6-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:38:44.712835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.712807    2582 generic.go:356] "Generic (PLEG): container finished" podID="57d6fe0b-727d-42da-8cbb-939697dbdf82" containerID="22fe737efd934da248fb75c114a56d793db3d72231a43e738e0acfb15f6422e8" exitCode=0
Jul 17 05:38:44.713285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.712899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj" event={"ID":"57d6fe0b-727d-42da-8cbb-939697dbdf82","Type":"ContainerDied","Data":"22fe737efd934da248fb75c114a56d793db3d72231a43e738e0acfb15f6422e8"}
Jul 17 05:38:44.713285 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.712955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj" event={"ID":"57d6fe0b-727d-42da-8cbb-939697dbdf82","Type":"ContainerStarted","Data":"d0bbcccaf0453c917d82b9af1db99846d40743b9f4a20c347cbe50fa53575239"}
Jul 17 05:38:44.714063 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.714045    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"
Jul 17 05:38:44.714170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.714061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd" event={"ID":"1c1be9f6-8673-4430-8f63-3af7c508dec6","Type":"ContainerDied","Data":"36d364d477ad4e7f00c683c39daee5d470352abeba8fb81b7e70829bf3f2f77e"}
Jul 17 05:38:44.714170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.714107    2582 scope.go:117] "RemoveContainer" containerID="42ed622ca893dd2041f05316788bdf4971a9a7e5ca92e2a8a9d6cd4e00ddaef4"
Jul 17 05:38:44.716413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.716393    2582 scope.go:117] "RemoveContainer" containerID="890017505d96c230937ab81f9cc5004cdf69e1d8ff20f5d832d9e66cd8904445"
Jul 17 05:38:44.716592 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:44.716574    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-2dxxt_odh-ai-gateway-infra(bccf528c-0888-42f4-bdf8-50f6bb80f174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174"
Jul 17 05:38:44.764765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.762334    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"]
Jul 17 05:38:44.766864 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:44.766841    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-lqnzd"]
Jul 17 05:38:45.850475 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:45.850449    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:45.869252 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:45.869222    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d6fe0b-727d-42da-8cbb-939697dbdf82-maas-api-service-ca\") pod \"57d6fe0b-727d-42da-8cbb-939697dbdf82\" (UID: \"57d6fe0b-727d-42da-8cbb-939697dbdf82\") "
Jul 17 05:38:45.869622 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:45.869599    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d6fe0b-727d-42da-8cbb-939697dbdf82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57d6fe0b-727d-42da-8cbb-939697dbdf82" (UID: "57d6fe0b-727d-42da-8cbb-939697dbdf82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:38:45.970239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:45.970202    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d6fe0b-727d-42da-8cbb-939697dbdf82-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:38:46.727067 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:46.727044    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj"
Jul 17 05:38:46.729937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:46.729911    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" path="/var/lib/kubelet/pods/1c1be9f6-8673-4430-8f63-3af7c508dec6/volumes"
Jul 17 05:38:46.730281 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:46.730249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj" event={"ID":"57d6fe0b-727d-42da-8cbb-939697dbdf82","Type":"ContainerDied","Data":"d0bbcccaf0453c917d82b9af1db99846d40743b9f4a20c347cbe50fa53575239"}
Jul 17 05:38:46.730344 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:46.730299    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0bbcccaf0453c917d82b9af1db99846d40743b9f4a20c347cbe50fa53575239"
Jul 17 05:38:57.441606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441573    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"]
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441885    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441897    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441904    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d6fe0b-727d-42da-8cbb-939697dbdf82" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441910    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d6fe0b-727d-42da-8cbb-939697dbdf82" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441924    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441929    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441975    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d6fe0b-727d-42da-8cbb-939697dbdf82" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441982    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441988    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.442053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.441993    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:38:57.446381 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.446361    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:38:57.454043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.453300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"]
Jul 17 05:38:57.546860 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.546825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5\" (UID: \"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:38:57.647489 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.647447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5\" (UID: \"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:38:57.648138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.648116    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5\" (UID: \"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:38:57.757810 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.757723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:38:57.897121 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:57.897072    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"]
Jul 17 05:38:57.900073 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:38:57.900047    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3162fa2_3f4b_4d4c_9f55_6fef84538ff7.slice/crio-c8ee600540b64e1344affc5bb03812e1ab98162b9cbf576b4e85a31dce87b381 WatchSource:0}: Error finding container c8ee600540b64e1344affc5bb03812e1ab98162b9cbf576b4e85a31dce87b381: Status 404 returned error can't find the container with id c8ee600540b64e1344affc5bb03812e1ab98162b9cbf576b4e85a31dce87b381
Jul 17 05:38:58.724721 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:58.724686    2582 scope.go:117] "RemoveContainer" containerID="890017505d96c230937ab81f9cc5004cdf69e1d8ff20f5d832d9e66cd8904445"
Jul 17 05:38:58.784096 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:58.784071    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerID="0a8fe5c2e05311af8f72b94c4448283abf823a255d9c014e15b25edd77ac770c" exitCode=6
Jul 17 05:38:58.784224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:58.784139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" event={"ID":"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7","Type":"ContainerDied","Data":"0a8fe5c2e05311af8f72b94c4448283abf823a255d9c014e15b25edd77ac770c"}
Jul 17 05:38:58.784224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:58.784182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" event={"ID":"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7","Type":"ContainerStarted","Data":"c8ee600540b64e1344affc5bb03812e1ab98162b9cbf576b4e85a31dce87b381"}
Jul 17 05:38:58.784333 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:58.784321    2582 scope.go:117] "RemoveContainer" containerID="0a8fe5c2e05311af8f72b94c4448283abf823a255d9c014e15b25edd77ac770c"
Jul 17 05:38:59.790136 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.790109    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerID="bb4b4aea1acf74bbfe112ac70009cb3ae0ccc46fc72bce7de51ea0dbc20d6319" exitCode=6
Jul 17 05:38:59.790631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.790191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" event={"ID":"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7","Type":"ContainerDied","Data":"bb4b4aea1acf74bbfe112ac70009cb3ae0ccc46fc72bce7de51ea0dbc20d6319"}
Jul 17 05:38:59.790631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.790272    2582 scope.go:117] "RemoveContainer" containerID="0a8fe5c2e05311af8f72b94c4448283abf823a255d9c014e15b25edd77ac770c"
Jul 17 05:38:59.790631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.790473    2582 scope.go:117] "RemoveContainer" containerID="bb4b4aea1acf74bbfe112ac70009cb3ae0ccc46fc72bce7de51ea0dbc20d6319"
Jul 17 05:38:59.790806 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:59.790713    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5_odh-ai-gateway-infra(f3162fa2-3f4b-4d4c-9f55-6fef84538ff7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7"
Jul 17 05:38:59.792138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.792119    2582 generic.go:356] "Generic (PLEG): container finished" podID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerID="eb0d330c0f46a432f4ff17f4612f39df84b6cef32f0249baf65619718ba25ea6" exitCode=6
Jul 17 05:38:59.792231 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.792166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" event={"ID":"bccf528c-0888-42f4-bdf8-50f6bb80f174","Type":"ContainerDied","Data":"eb0d330c0f46a432f4ff17f4612f39df84b6cef32f0249baf65619718ba25ea6"}
Jul 17 05:38:59.792457 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.792444    2582 scope.go:117] "RemoveContainer" containerID="eb0d330c0f46a432f4ff17f4612f39df84b6cef32f0249baf65619718ba25ea6"
Jul 17 05:38:59.792633 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:38:59.792608    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-2dxxt_odh-ai-gateway-infra(bccf528c-0888-42f4-bdf8-50f6bb80f174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174"
Jul 17 05:38:59.800845 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:38:59.800822    2582 scope.go:117] "RemoveContainer" containerID="890017505d96c230937ab81f9cc5004cdf69e1d8ff20f5d832d9e66cd8904445"
Jul 17 05:39:00.799895 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:00.799870    2582 scope.go:117] "RemoveContainer" containerID="bb4b4aea1acf74bbfe112ac70009cb3ae0ccc46fc72bce7de51ea0dbc20d6319"
Jul 17 05:39:00.800368 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:00.800067    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5_odh-ai-gateway-infra(f3162fa2-3f4b-4d4c-9f55-6fef84538ff7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7"
Jul 17 05:39:00.833728 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:00.833683    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"]
Jul 17 05:39:00.969465 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:00.969437    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:39:01.072696 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.072588    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccf528c-0888-42f4-bdf8-50f6bb80f174-maas-api-service-ca\") pod \"bccf528c-0888-42f4-bdf8-50f6bb80f174\" (UID: \"bccf528c-0888-42f4-bdf8-50f6bb80f174\") "
Jul 17 05:39:01.072979 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.072954    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccf528c-0888-42f4-bdf8-50f6bb80f174-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bccf528c-0888-42f4-bdf8-50f6bb80f174" (UID: "bccf528c-0888-42f4-bdf8-50f6bb80f174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:01.173581 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.173536    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccf528c-0888-42f4-bdf8-50f6bb80f174-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:01.806584 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.806553    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"
Jul 17 05:39:01.807046 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.806583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt" event={"ID":"bccf528c-0888-42f4-bdf8-50f6bb80f174","Type":"ContainerDied","Data":"4328a56024af83b7f56dbbd13019d3a5cdb896d601bfe40ae55244c284824224"}
Jul 17 05:39:01.807046 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.806642    2582 scope.go:117] "RemoveContainer" containerID="eb0d330c0f46a432f4ff17f4612f39df84b6cef32f0249baf65619718ba25ea6"
Jul 17 05:39:01.828476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.828454    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"]
Jul 17 05:39:01.832372 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:01.832352    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-2dxxt"]
Jul 17 05:39:02.729910 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:02.729879    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" path="/var/lib/kubelet/pods/bccf528c-0888-42f4-bdf8-50f6bb80f174/volumes"
Jul 17 05:39:09.347273 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347234    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"]
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347547    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347559    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1be9f6-8673-4430-8f63-3af7c508dec6" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347569    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347575    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347584    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347589    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347601    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347606    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347650    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347970 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347696    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.347970 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.347704    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccf528c-0888-42f4-bdf8-50f6bb80f174" containerName="revoke-keys"
Jul 17 05:39:09.350446 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.350425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:09.358537 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.358508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"]
Jul 17 05:39:09.418827 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.418794    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"]
Jul 17 05:39:09.422798 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.422767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:09.429139 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.429061    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"]
Jul 17 05:39:09.436310 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.436251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70986f-8baa-4529-928a-c4da699b15e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz\" (UID: \"9a70986f-8baa-4529-928a-c4da699b15e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:09.436434 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.436340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw\" (UID: \"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:09.537721 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.537680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70986f-8baa-4529-928a-c4da699b15e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz\" (UID: \"9a70986f-8baa-4529-928a-c4da699b15e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:09.537911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.537728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw\" (UID: \"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:09.538387 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.538370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw\" (UID: \"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:09.538436 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.538370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70986f-8baa-4529-928a-c4da699b15e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz\" (UID: \"9a70986f-8baa-4529-928a-c4da699b15e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:09.661598 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.661521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:09.736747 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.736720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:09.799546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.798384    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"]
Jul 17 05:39:09.842414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.842388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" event={"ID":"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4","Type":"ContainerStarted","Data":"e18f06469dd64fc9ed0318ae1205118af7e7e9125b15a686e4045302c199ede9"}
Jul 17 05:39:09.902425 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:09.902361    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"]
Jul 17 05:39:09.904403 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:39:09.904380    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70986f_8baa_4529_928a_c4da699b15e4.slice/crio-889aa0d4759bc90f7c6b847232ba4fb8de7c53a0271a78686039fb474bba9c50 WatchSource:0}: Error finding container 889aa0d4759bc90f7c6b847232ba4fb8de7c53a0271a78686039fb474bba9c50: Status 404 returned error can't find the container with id 889aa0d4759bc90f7c6b847232ba4fb8de7c53a0271a78686039fb474bba9c50
Jul 17 05:39:10.848319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:10.848199    2582 generic.go:356] "Generic (PLEG): container finished" podID="9a70986f-8baa-4529-928a-c4da699b15e4" containerID="e362b4e2c8319b94bf0821e3bf8107dff7f1f1fe855f9db91ff3d4e7a6ae4190" exitCode=0
Jul 17 05:39:10.848319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:10.848288    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz" event={"ID":"9a70986f-8baa-4529-928a-c4da699b15e4","Type":"ContainerDied","Data":"e362b4e2c8319b94bf0821e3bf8107dff7f1f1fe855f9db91ff3d4e7a6ae4190"}
Jul 17 05:39:10.848811 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:10.848350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz" event={"ID":"9a70986f-8baa-4529-928a-c4da699b15e4","Type":"ContainerStarted","Data":"889aa0d4759bc90f7c6b847232ba4fb8de7c53a0271a78686039fb474bba9c50"}
Jul 17 05:39:10.849820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:10.849798    2582 generic.go:356] "Generic (PLEG): container finished" podID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerID="47389d3a1bcefdffcd252b64766c48ffa15b2bd9b1bea2eb5bdbc908ffad0028" exitCode=6
Jul 17 05:39:10.849898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:10.849828    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" event={"ID":"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4","Type":"ContainerDied","Data":"47389d3a1bcefdffcd252b64766c48ffa15b2bd9b1bea2eb5bdbc908ffad0028"}
Jul 17 05:39:10.850044 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:10.850031    2582 scope.go:117] "RemoveContainer" containerID="47389d3a1bcefdffcd252b64766c48ffa15b2bd9b1bea2eb5bdbc908ffad0028"
Jul 17 05:39:11.856563 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:11.856530    2582 generic.go:356] "Generic (PLEG): container finished" podID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerID="28f273977553d72db63848be0f3700bd4eb075dc5f58d3112c43d7a38dc22d2f" exitCode=6
Jul 17 05:39:11.857053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:11.856619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" event={"ID":"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4","Type":"ContainerDied","Data":"28f273977553d72db63848be0f3700bd4eb075dc5f58d3112c43d7a38dc22d2f"}
Jul 17 05:39:11.857053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:11.856683    2582 scope.go:117] "RemoveContainer" containerID="47389d3a1bcefdffcd252b64766c48ffa15b2bd9b1bea2eb5bdbc908ffad0028"
Jul 17 05:39:11.857053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:11.856852    2582 scope.go:117] "RemoveContainer" containerID="28f273977553d72db63848be0f3700bd4eb075dc5f58d3112c43d7a38dc22d2f"
Jul 17 05:39:11.857456 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:11.857100    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw_odh-ai-gateway-infra(ebf68191-6bbe-40f1-8ac5-71e325f0c6e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4"
Jul 17 05:39:11.987194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:11.987170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:12.057176 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.057142    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70986f-8baa-4529-928a-c4da699b15e4-maas-api-service-ca\") pod \"9a70986f-8baa-4529-928a-c4da699b15e4\" (UID: \"9a70986f-8baa-4529-928a-c4da699b15e4\") "
Jul 17 05:39:12.057541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.057517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a70986f-8baa-4529-928a-c4da699b15e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a70986f-8baa-4529-928a-c4da699b15e4" (UID: "9a70986f-8baa-4529-928a-c4da699b15e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:12.157722 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.157632    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a70986f-8baa-4529-928a-c4da699b15e4-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:12.862704 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.862672    2582 scope.go:117] "RemoveContainer" containerID="28f273977553d72db63848be0f3700bd4eb075dc5f58d3112c43d7a38dc22d2f"
Jul 17 05:39:12.863168 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:12.862879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw_odh-ai-gateway-infra(ebf68191-6bbe-40f1-8ac5-71e325f0c6e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4"
Jul 17 05:39:12.863736 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.863716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz" event={"ID":"9a70986f-8baa-4529-928a-c4da699b15e4","Type":"ContainerDied","Data":"889aa0d4759bc90f7c6b847232ba4fb8de7c53a0271a78686039fb474bba9c50"}
Jul 17 05:39:12.863803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.863733    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz"
Jul 17 05:39:12.863803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:12.863756    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="889aa0d4759bc90f7c6b847232ba4fb8de7c53a0271a78686039fb474bba9c50"
Jul 17 05:39:15.724700 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:15.724668    2582 scope.go:117] "RemoveContainer" containerID="bb4b4aea1acf74bbfe112ac70009cb3ae0ccc46fc72bce7de51ea0dbc20d6319"
Jul 17 05:39:16.885371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:16.885289    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerID="ef0358c3ba15f89f39ab866478b6b04f953bb180b51c94a30b99821e61f61529" exitCode=6
Jul 17 05:39:16.885790 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:16.885367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" event={"ID":"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7","Type":"ContainerDied","Data":"ef0358c3ba15f89f39ab866478b6b04f953bb180b51c94a30b99821e61f61529"}
Jul 17 05:39:16.885790 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:16.885431    2582 scope.go:117] "RemoveContainer" containerID="bb4b4aea1acf74bbfe112ac70009cb3ae0ccc46fc72bce7de51ea0dbc20d6319"
Jul 17 05:39:16.885790 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:16.885695    2582 scope.go:117] "RemoveContainer" containerID="ef0358c3ba15f89f39ab866478b6b04f953bb180b51c94a30b99821e61f61529"
Jul 17 05:39:16.885936 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:16.885908    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5_odh-ai-gateway-infra(f3162fa2-3f4b-4d4c-9f55-6fef84538ff7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7"
Jul 17 05:39:17.913158 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:17.913124    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"]
Jul 17 05:39:18.046884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.046858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:39:18.108270 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.108237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7-maas-api-service-ca\") pod \"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7\" (UID: \"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7\") "
Jul 17 05:39:18.108587 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.108566    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" (UID: "f3162fa2-3f4b-4d4c-9f55-6fef84538ff7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:18.209709 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.209678    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:18.897677 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.897648    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"
Jul 17 05:39:18.897841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.897645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5" event={"ID":"f3162fa2-3f4b-4d4c-9f55-6fef84538ff7","Type":"ContainerDied","Data":"c8ee600540b64e1344affc5bb03812e1ab98162b9cbf576b4e85a31dce87b381"}
Jul 17 05:39:18.897841 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.897784    2582 scope.go:117] "RemoveContainer" containerID="ef0358c3ba15f89f39ab866478b6b04f953bb180b51c94a30b99821e61f61529"
Jul 17 05:39:18.914580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.914544    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"]
Jul 17 05:39:18.921994 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:18.921965    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-qc5w5"]
Jul 17 05:39:19.877019 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:19.876982    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"]
Jul 17 05:39:19.877269 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:19.877207    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68" podUID="36114828-85bc-452c-afbc-b84e0763a16a" containerName="maas-api" containerID="cri-o://f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba" gracePeriod=30
Jul 17 05:39:20.119427 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.119405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:39:20.226420 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.226382    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36114828-85bc-452c-afbc-b84e0763a16a-maas-api-tls\") pod \"36114828-85bc-452c-afbc-b84e0763a16a\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") "
Jul 17 05:39:20.226628 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.226454    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36114828-85bc-452c-afbc-b84e0763a16a-openshift-service-ca\") pod \"36114828-85bc-452c-afbc-b84e0763a16a\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") "
Jul 17 05:39:20.226628 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.226475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7svs8\" (UniqueName: \"kubernetes.io/projected/36114828-85bc-452c-afbc-b84e0763a16a-kube-api-access-7svs8\") pod \"36114828-85bc-452c-afbc-b84e0763a16a\" (UID: \"36114828-85bc-452c-afbc-b84e0763a16a\") "
Jul 17 05:39:20.226879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.226850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36114828-85bc-452c-afbc-b84e0763a16a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36114828-85bc-452c-afbc-b84e0763a16a" (UID: "36114828-85bc-452c-afbc-b84e0763a16a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:20.228836 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.228810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36114828-85bc-452c-afbc-b84e0763a16a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36114828-85bc-452c-afbc-b84e0763a16a" (UID: "36114828-85bc-452c-afbc-b84e0763a16a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:39:20.228925 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.228831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36114828-85bc-452c-afbc-b84e0763a16a-kube-api-access-7svs8" (OuterVolumeSpecName: "kube-api-access-7svs8") pod "36114828-85bc-452c-afbc-b84e0763a16a" (UID: "36114828-85bc-452c-afbc-b84e0763a16a"). InnerVolumeSpecName "kube-api-access-7svs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:39:20.327534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.327488    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36114828-85bc-452c-afbc-b84e0763a16a-maas-api-tls\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:20.327534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.327530    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36114828-85bc-452c-afbc-b84e0763a16a-openshift-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:20.327534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.327540    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svs8\" (UniqueName: \"kubernetes.io/projected/36114828-85bc-452c-afbc-b84e0763a16a-kube-api-access-7svs8\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:20.730037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.730002    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" path="/var/lib/kubelet/pods/f3162fa2-3f4b-4d4c-9f55-6fef84538ff7/volumes"
Jul 17 05:39:20.909054 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.909025    2582 generic.go:356] "Generic (PLEG): container finished" podID="36114828-85bc-452c-afbc-b84e0763a16a" containerID="f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba" exitCode=0
Jul 17 05:39:20.909218 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.909074    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"
Jul 17 05:39:20.909218 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.909117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68" event={"ID":"36114828-85bc-452c-afbc-b84e0763a16a","Type":"ContainerDied","Data":"f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba"}
Jul 17 05:39:20.909218 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.909167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68" event={"ID":"36114828-85bc-452c-afbc-b84e0763a16a","Type":"ContainerDied","Data":"e0f6e26a48efb34f3c4fb7f6afba84877c37904665109d3f826a6f8622ce9ddb"}
Jul 17 05:39:20.909218 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.909191    2582 scope.go:117] "RemoveContainer" containerID="f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba"
Jul 17 05:39:20.917541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.917523    2582 scope.go:117] "RemoveContainer" containerID="f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba"
Jul 17 05:39:20.917825 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:20.917808    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba\": container with ID starting with f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba not found: ID does not exist" containerID="f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba"
Jul 17 05:39:20.917930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.917836    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba"} err="failed to get container status \"f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba\": rpc error: code = NotFound desc = could not find container \"f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba\": container with ID starting with f4c623b64b21c812e191fece86bd496357f9377d3f11aecabea88af6f876d1ba not found: ID does not exist"
Jul 17 05:39:20.927268 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.927236    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"]
Jul 17 05:39:20.930736 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:20.930710    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ce2966-6df868f4b6-jnk68"]
Jul 17 05:39:22.730051 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:22.730017    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36114828-85bc-452c-afbc-b84e0763a16a" path="/var/lib/kubelet/pods/36114828-85bc-452c-afbc-b84e0763a16a/volumes"
Jul 17 05:39:24.769634 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769597    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"]
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769905    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a70986f-8baa-4529-928a-c4da699b15e4" containerName="revoke-keys"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769916    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a70986f-8baa-4529-928a-c4da699b15e4" containerName="revoke-keys"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769926    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769931    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769948    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769953    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769961    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36114828-85bc-452c-afbc-b84e0763a16a" containerName="maas-api"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.769966    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="36114828-85bc-452c-afbc-b84e0763a16a" containerName="maas-api"
Jul 17 05:39:24.770012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.770013    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:24.770343 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.770023    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:24.770343 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.770028    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a70986f-8baa-4529-928a-c4da699b15e4" containerName="revoke-keys"
Jul 17 05:39:24.770343 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.770034    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="36114828-85bc-452c-afbc-b84e0763a16a" containerName="maas-api"
Jul 17 05:39:24.774361 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.774342    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:24.782474 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.782444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"]
Jul 17 05:39:24.866451 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.866407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c8ae5c-9207-4fb4-8d06-919b287a0dff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk\" (UID: \"53c8ae5c-9207-4fb4-8d06-919b287a0dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:24.967749 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.967699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c8ae5c-9207-4fb4-8d06-919b287a0dff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk\" (UID: \"53c8ae5c-9207-4fb4-8d06-919b287a0dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:24.968413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:24.968394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c8ae5c-9207-4fb4-8d06-919b287a0dff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk\" (UID: \"53c8ae5c-9207-4fb4-8d06-919b287a0dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:25.085159 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:25.085069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:25.213852 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:25.213825    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"]
Jul 17 05:39:25.216200 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:39:25.216173    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c8ae5c_9207_4fb4_8d06_919b287a0dff.slice/crio-0ead400fd16e78a799410eb246e618552d3ffe4f01a210e5b12cb595deed7990 WatchSource:0}: Error finding container 0ead400fd16e78a799410eb246e618552d3ffe4f01a210e5b12cb595deed7990: Status 404 returned error can't find the container with id 0ead400fd16e78a799410eb246e618552d3ffe4f01a210e5b12cb595deed7990
Jul 17 05:39:25.935405 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:25.935372    2582 generic.go:356] "Generic (PLEG): container finished" podID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerID="6fa62956d3d19ac236f061dfb1256fa584a9e453535bcb89ceeba26d3944c551" exitCode=6
Jul 17 05:39:25.935789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:25.935407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" event={"ID":"53c8ae5c-9207-4fb4-8d06-919b287a0dff","Type":"ContainerDied","Data":"6fa62956d3d19ac236f061dfb1256fa584a9e453535bcb89ceeba26d3944c551"}
Jul 17 05:39:25.935789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:25.935467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" event={"ID":"53c8ae5c-9207-4fb4-8d06-919b287a0dff","Type":"ContainerStarted","Data":"0ead400fd16e78a799410eb246e618552d3ffe4f01a210e5b12cb595deed7990"}
Jul 17 05:39:25.935789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:25.935661    2582 scope.go:117] "RemoveContainer" containerID="6fa62956d3d19ac236f061dfb1256fa584a9e453535bcb89ceeba26d3944c551"
Jul 17 05:39:26.726698 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:26.726670    2582 scope.go:117] "RemoveContainer" containerID="28f273977553d72db63848be0f3700bd4eb075dc5f58d3112c43d7a38dc22d2f"
Jul 17 05:39:26.941602 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:26.941565    2582 generic.go:356] "Generic (PLEG): container finished" podID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerID="b8dcb8ce31762b1faafc6b7a53ee7ed508c8612bfb4e96a77bbe7afa2bc8988c" exitCode=6
Jul 17 05:39:26.942007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:26.941633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" event={"ID":"53c8ae5c-9207-4fb4-8d06-919b287a0dff","Type":"ContainerDied","Data":"b8dcb8ce31762b1faafc6b7a53ee7ed508c8612bfb4e96a77bbe7afa2bc8988c"}
Jul 17 05:39:26.942007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:26.941686    2582 scope.go:117] "RemoveContainer" containerID="6fa62956d3d19ac236f061dfb1256fa584a9e453535bcb89ceeba26d3944c551"
Jul 17 05:39:26.942007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:26.941853    2582 scope.go:117] "RemoveContainer" containerID="b8dcb8ce31762b1faafc6b7a53ee7ed508c8612bfb4e96a77bbe7afa2bc8988c"
Jul 17 05:39:26.942129 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:26.942050    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk_odh-ai-gateway-infra(53c8ae5c-9207-4fb4-8d06-919b287a0dff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff"
Jul 17 05:39:27.948121 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:27.948093    2582 generic.go:356] "Generic (PLEG): container finished" podID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerID="ac7b327c06f395fb5998903cd98625a5dcc697a35b8fd0cb5d1f886fd79fb579" exitCode=6
Jul 17 05:39:27.948643 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:27.948161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" event={"ID":"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4","Type":"ContainerDied","Data":"ac7b327c06f395fb5998903cd98625a5dcc697a35b8fd0cb5d1f886fd79fb579"}
Jul 17 05:39:27.948643 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:27.948220    2582 scope.go:117] "RemoveContainer" containerID="28f273977553d72db63848be0f3700bd4eb075dc5f58d3112c43d7a38dc22d2f"
Jul 17 05:39:27.948643 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:27.948527    2582 scope.go:117] "RemoveContainer" containerID="ac7b327c06f395fb5998903cd98625a5dcc697a35b8fd0cb5d1f886fd79fb579"
Jul 17 05:39:27.948816 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:27.948740    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw_odh-ai-gateway-infra(ebf68191-6bbe-40f1-8ac5-71e325f0c6e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4"
Jul 17 05:39:27.950049 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:27.950032    2582 scope.go:117] "RemoveContainer" containerID="b8dcb8ce31762b1faafc6b7a53ee7ed508c8612bfb4e96a77bbe7afa2bc8988c"
Jul 17 05:39:27.950195 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:27.950178    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk_odh-ai-gateway-infra(53c8ae5c-9207-4fb4-8d06-919b287a0dff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff"
Jul 17 05:39:28.974534 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:28.974501    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"]
Jul 17 05:39:29.109350 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.109328    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:29.207743 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.207707    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4-maas-api-service-ca\") pod \"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4\" (UID: \"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4\") "
Jul 17 05:39:29.208097 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.208071    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" (UID: "ebf68191-6bbe-40f1-8ac5-71e325f0c6e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:29.308888 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.308803    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:29.961185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.961158    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"
Jul 17 05:39:29.961382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.961151    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw" event={"ID":"ebf68191-6bbe-40f1-8ac5-71e325f0c6e4","Type":"ContainerDied","Data":"e18f06469dd64fc9ed0318ae1205118af7e7e9125b15a686e4045302c199ede9"}
Jul 17 05:39:29.961382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.961308    2582 scope.go:117] "RemoveContainer" containerID="ac7b327c06f395fb5998903cd98625a5dcc697a35b8fd0cb5d1f886fd79fb579"
Jul 17 05:39:29.981546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.981521    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"]
Jul 17 05:39:29.987196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:29.987173    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-smpxw"]
Jul 17 05:39:30.729917 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:30.729886    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" path="/var/lib/kubelet/pods/ebf68191-6bbe-40f1-8ac5-71e325f0c6e4/volumes"
Jul 17 05:39:39.036960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.036927    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"]
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037241    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037253    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037283    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037291    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037309    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037315    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037325    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037330    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037385    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3162fa2-3f4b-4d4c-9f55-6fef84538ff7" containerName="revoke-keys"
Jul 17 05:39:39.037617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037393    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037399    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.037617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.037406    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf68191-6bbe-40f1-8ac5-71e325f0c6e4" containerName="revoke-keys"
Jul 17 05:39:39.040090 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.040073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.043157 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.043135    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xwcd7\""
Jul 17 05:39:39.043397 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.043381    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-560f5b-serving-cert\""
Jul 17 05:39:39.044304 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.044289    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 05:39:39.052757 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.052730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"]
Jul 17 05:39:39.196467 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.196426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f138551-bf8a-464a-acd5-ade1e2c84f9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.196692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.196485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdd2l\" (UniqueName: \"kubernetes.io/projected/7f138551-bf8a-464a-acd5-ade1e2c84f9d-kube-api-access-rdd2l\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.196692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.196557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f138551-bf8a-464a-acd5-ade1e2c84f9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.297391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.297282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdd2l\" (UniqueName: \"kubernetes.io/projected/7f138551-bf8a-464a-acd5-ade1e2c84f9d-kube-api-access-rdd2l\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.297391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.297358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f138551-bf8a-464a-acd5-ade1e2c84f9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.297391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.297384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f138551-bf8a-464a-acd5-ade1e2c84f9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.297973 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.297951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f138551-bf8a-464a-acd5-ade1e2c84f9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.299864 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.299847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f138551-bf8a-464a-acd5-ade1e2c84f9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.306984 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.306963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdd2l\" (UniqueName: \"kubernetes.io/projected/7f138551-bf8a-464a-acd5-ade1e2c84f9d-kube-api-access-rdd2l\") pod \"maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.350025 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.349986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:39.487151 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:39.487115    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"]
Jul 17 05:39:39.490792 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:39:39.490758    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f138551_bf8a_464a_acd5_ade1e2c84f9d.slice/crio-c3360203c2a179b16d3f11380df2331e983899cd81f504f226f638ee34afda66 WatchSource:0}: Error finding container c3360203c2a179b16d3f11380df2331e983899cd81f504f226f638ee34afda66: Status 404 returned error can't find the container with id c3360203c2a179b16d3f11380df2331e983899cd81f504f226f638ee34afda66
Jul 17 05:39:40.008197 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:40.008163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh" event={"ID":"7f138551-bf8a-464a-acd5-ade1e2c84f9d","Type":"ContainerStarted","Data":"c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915"}
Jul 17 05:39:40.008384 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:40.008214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh" event={"ID":"7f138551-bf8a-464a-acd5-ade1e2c84f9d","Type":"ContainerStarted","Data":"c3360203c2a179b16d3f11380df2331e983899cd81f504f226f638ee34afda66"}
Jul 17 05:39:40.008384 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:40.008246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:40.029243 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:40.029193    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh" podStartSLOduration=0.626521659 podStartE2EDuration="1.029173634s" totalImagesPullingTime="402.651975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:39:40.028840557 +0000 UTC m=+2983.765902894" watchObservedRunningTime="2026-07-17 05:39:40.029173634 +0000 UTC m=+2983.766235970"
Jul 17 05:39:41.232397 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.232359    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"]
Jul 17 05:39:41.235543 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.235521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:41.244193 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.244168    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"]
Jul 17 05:39:41.313744 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.313707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2873e9-55a0-40de-8cab-4265abf297cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-hhbc8\" (UID: \"4f2873e9-55a0-40de-8cab-4265abf297cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:41.415072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.415033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2873e9-55a0-40de-8cab-4265abf297cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-hhbc8\" (UID: \"4f2873e9-55a0-40de-8cab-4265abf297cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:41.415712 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.415694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2873e9-55a0-40de-8cab-4265abf297cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-hhbc8\" (UID: \"4f2873e9-55a0-40de-8cab-4265abf297cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:41.547348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.547205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:41.683178 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:41.683040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"]
Jul 17 05:39:41.685670 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:39:41.685647    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2873e9_55a0_40de_8cab_4265abf297cd.slice/crio-ef6adb143e52be59b8c54576f69e43af40d6dbfc8a9fa16d6447d509aece0a3a WatchSource:0}: Error finding container ef6adb143e52be59b8c54576f69e43af40d6dbfc8a9fa16d6447d509aece0a3a: Status 404 returned error can't find the container with id ef6adb143e52be59b8c54576f69e43af40d6dbfc8a9fa16d6447d509aece0a3a
Jul 17 05:39:42.018331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:42.018293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerStarted","Data":"cf81aaf35010a57ec5c61df4ad673db67caa2540aa2903189c88146f21fada95"}
Jul 17 05:39:42.018522 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:42.018382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerStarted","Data":"ef6adb143e52be59b8c54576f69e43af40d6dbfc8a9fa16d6447d509aece0a3a"}
Jul 17 05:39:42.034338 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:42.034284    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" podStartSLOduration=1.034252336 podStartE2EDuration="1.034252336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:39:42.033582068 +0000 UTC m=+2985.770644426" watchObservedRunningTime="2026-07-17 05:39:42.034252336 +0000 UTC m=+2985.771314672"
Jul 17 05:39:42.725351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:42.725319    2582 scope.go:117] "RemoveContainer" containerID="b8dcb8ce31762b1faafc6b7a53ee7ed508c8612bfb4e96a77bbe7afa2bc8988c"
Jul 17 05:39:43.024311 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:43.024278    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerID="cf81aaf35010a57ec5c61df4ad673db67caa2540aa2903189c88146f21fada95" exitCode=6
Jul 17 05:39:43.024472 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:43.024355    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerDied","Data":"cf81aaf35010a57ec5c61df4ad673db67caa2540aa2903189c88146f21fada95"}
Jul 17 05:39:43.024569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:43.024552    2582 scope.go:117] "RemoveContainer" containerID="cf81aaf35010a57ec5c61df4ad673db67caa2540aa2903189c88146f21fada95"
Jul 17 05:39:43.026279 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:43.026236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" event={"ID":"53c8ae5c-9207-4fb4-8d06-919b287a0dff","Type":"ContainerStarted","Data":"3f448286ff91a59fb97fad720d0e5f1ff1caafe591f550c2ac3aec78d363e903"}
Jul 17 05:39:43.056173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:43.056120    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" podStartSLOduration=19.056103977 podStartE2EDuration="19.056103977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:39:43.054427701 +0000 UTC m=+2986.791490038" watchObservedRunningTime="2026-07-17 05:39:43.056103977 +0000 UTC m=+2986.793166313"
Jul 17 05:39:44.032805 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.032767    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerID="1b44af1e4738ed8d905124f6be1d28e64821ce7378a7754267878a670812d734" exitCode=6
Jul 17 05:39:44.032805 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.032799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerDied","Data":"1b44af1e4738ed8d905124f6be1d28e64821ce7378a7754267878a670812d734"}
Jul 17 05:39:44.033370 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.032858    2582 scope.go:117] "RemoveContainer" containerID="cf81aaf35010a57ec5c61df4ad673db67caa2540aa2903189c88146f21fada95"
Jul 17 05:39:44.033370 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.033126    2582 scope.go:117] "RemoveContainer" containerID="1b44af1e4738ed8d905124f6be1d28e64821ce7378a7754267878a670812d734"
Jul 17 05:39:44.033370 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:44.033351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-hhbc8_odh-ai-gateway-infra(4f2873e9-55a0-40de-8cab-4265abf297cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd"
Jul 17 05:39:44.034906 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.034887    2582 generic.go:356] "Generic (PLEG): container finished" podID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerID="3f448286ff91a59fb97fad720d0e5f1ff1caafe591f550c2ac3aec78d363e903" exitCode=6
Jul 17 05:39:44.035016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.034921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" event={"ID":"53c8ae5c-9207-4fb4-8d06-919b287a0dff","Type":"ContainerDied","Data":"3f448286ff91a59fb97fad720d0e5f1ff1caafe591f550c2ac3aec78d363e903"}
Jul 17 05:39:44.035151 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.035139    2582 scope.go:117] "RemoveContainer" containerID="3f448286ff91a59fb97fad720d0e5f1ff1caafe591f550c2ac3aec78d363e903"
Jul 17 05:39:44.035314 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:44.035297    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk_odh-ai-gateway-infra(53c8ae5c-9207-4fb4-8d06-919b287a0dff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff"
Jul 17 05:39:44.043976 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.043959    2582 scope.go:117] "RemoveContainer" containerID="b8dcb8ce31762b1faafc6b7a53ee7ed508c8612bfb4e96a77bbe7afa2bc8988c"
Jul 17 05:39:44.077099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:44.077062    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"]
Jul 17 05:39:45.041113 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:45.041085    2582 scope.go:117] "RemoveContainer" containerID="1b44af1e4738ed8d905124f6be1d28e64821ce7378a7754267878a670812d734"
Jul 17 05:39:45.041603 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:45.041318    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-hhbc8_odh-ai-gateway-infra(4f2873e9-55a0-40de-8cab-4265abf297cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd"
Jul 17 05:39:45.182998 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:45.182964    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:45.344168 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:45.344067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c8ae5c-9207-4fb4-8d06-919b287a0dff-maas-api-service-ca\") pod \"53c8ae5c-9207-4fb4-8d06-919b287a0dff\" (UID: \"53c8ae5c-9207-4fb4-8d06-919b287a0dff\") "
Jul 17 05:39:45.344471 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:45.344448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c8ae5c-9207-4fb4-8d06-919b287a0dff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53c8ae5c-9207-4fb4-8d06-919b287a0dff" (UID: "53c8ae5c-9207-4fb4-8d06-919b287a0dff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:45.444819 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:45.444784    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c8ae5c-9207-4fb4-8d06-919b287a0dff-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:46.018177 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.018148    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:39:46.051416 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.051379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk" event={"ID":"53c8ae5c-9207-4fb4-8d06-919b287a0dff","Type":"ContainerDied","Data":"0ead400fd16e78a799410eb246e618552d3ffe4f01a210e5b12cb595deed7990"}
Jul 17 05:39:46.051937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.051457    2582 scope.go:117] "RemoveContainer" containerID="3f448286ff91a59fb97fad720d0e5f1ff1caafe591f550c2ac3aec78d363e903"
Jul 17 05:39:46.051937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.051484    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"
Jul 17 05:39:46.077214 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.077181    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"]
Jul 17 05:39:46.081221 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.081196    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-jstwk"]
Jul 17 05:39:46.735129 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:46.735097    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" path="/var/lib/kubelet/pods/53c8ae5c-9207-4fb4-8d06-919b287a0dff/volumes"
Jul 17 05:39:47.171465 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171372    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"]
Jul 17 05:39:47.171920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171810    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.171920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171829    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.171920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171847    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.171920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171856    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.171920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171893    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.171920 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.171902    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.172337 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.172000    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.172337 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.172015    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.172337 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.172027    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c8ae5c-9207-4fb4-8d06-919b287a0dff" containerName="revoke-keys"
Jul 17 05:39:47.176395 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.176374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.179148 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.179127    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-823cda-serving-cert\""
Jul 17 05:39:47.187202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.187177    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"]
Jul 17 05:39:47.259445 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.259406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.259627 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.259460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-openshift-service-ca\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.259627 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.259478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr22n\" (UniqueName: \"kubernetes.io/projected/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-kube-api-access-zr22n\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.360183 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.360143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-openshift-service-ca\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.360183 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.360193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr22n\" (UniqueName: \"kubernetes.io/projected/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-kube-api-access-zr22n\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.360468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.360294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.360468 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:47.360412    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-serving-cert: secret "maas-api-e2e-shared-b-823cda-serving-cert" not found
Jul 17 05:39:47.360569 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:47.360487    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls podName:45a50ce9-2dfc-46eb-aaef-ea0d195ae141 nodeName:}" failed. No retries permitted until 2026-07-17 05:39:47.86046631 +0000 UTC m=+2991.597528642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls") pod "maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" (UID: "45a50ce9-2dfc-46eb-aaef-ea0d195ae141") : secret "maas-api-e2e-shared-b-823cda-serving-cert" not found
Jul 17 05:39:47.360990 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.360968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-openshift-service-ca\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.372465 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.372430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr22n\" (UniqueName: \"kubernetes.io/projected/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-kube-api-access-zr22n\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.865803 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.865764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:47.868444 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:47.868421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls\") pod \"maas-api-e2e-shared-b-823cda-858f6754d5-dclb7\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:48.087743 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:48.087705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:48.224713 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:48.224686    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"]
Jul 17 05:39:48.227087 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:39:48.227061    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a50ce9_2dfc_46eb_aaef_ea0d195ae141.slice/crio-3d26f66b7bc8304f062bf67dc3b69df37dc62587023575a555e37cde7732ad2a WatchSource:0}: Error finding container 3d26f66b7bc8304f062bf67dc3b69df37dc62587023575a555e37cde7732ad2a: Status 404 returned error can't find the container with id 3d26f66b7bc8304f062bf67dc3b69df37dc62587023575a555e37cde7732ad2a
Jul 17 05:39:49.069231 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:49.069199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" event={"ID":"45a50ce9-2dfc-46eb-aaef-ea0d195ae141","Type":"ContainerStarted","Data":"036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256"}
Jul 17 05:39:49.069452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:49.069245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" event={"ID":"45a50ce9-2dfc-46eb-aaef-ea0d195ae141","Type":"ContainerStarted","Data":"3d26f66b7bc8304f062bf67dc3b69df37dc62587023575a555e37cde7732ad2a"}
Jul 17 05:39:49.069452 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:49.069295    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:49.088130 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:49.088082    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" podStartSLOduration=1.6725051309999999 podStartE2EDuration="2.088066892s" totalImagesPullingTime="415.561761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:39:49.086699065 +0000 UTC m=+2992.823761416" watchObservedRunningTime="2026-07-17 05:39:49.088066892 +0000 UTC m=+2992.825129228"
Jul 17 05:39:55.079413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:55.079379    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:39:55.725097 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:55.725065    2582 scope.go:117] "RemoveContainer" containerID="1b44af1e4738ed8d905124f6be1d28e64821ce7378a7754267878a670812d734"
Jul 17 05:39:56.108947 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:56.108916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerStarted","Data":"f26c2b56ec260dd7d77a7f1ac928c6735e1aff3271122e2de9697d66fa957182"}
Jul 17 05:39:56.863005 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:56.862980    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:39:56.869698 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:56.869675    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:39:57.120185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.120102    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerID="f26c2b56ec260dd7d77a7f1ac928c6735e1aff3271122e2de9697d66fa957182" exitCode=6
Jul 17 05:39:57.120185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.120159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerDied","Data":"f26c2b56ec260dd7d77a7f1ac928c6735e1aff3271122e2de9697d66fa957182"}
Jul 17 05:39:57.120694 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.120234    2582 scope.go:117] "RemoveContainer" containerID="1b44af1e4738ed8d905124f6be1d28e64821ce7378a7754267878a670812d734"
Jul 17 05:39:57.120694 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.120470    2582 scope.go:117] "RemoveContainer" containerID="f26c2b56ec260dd7d77a7f1ac928c6735e1aff3271122e2de9697d66fa957182"
Jul 17 05:39:57.120694 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:39:57.120676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-hhbc8_odh-ai-gateway-infra(4f2873e9-55a0-40de-8cab-4265abf297cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd"
Jul 17 05:39:57.145391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.145361    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"]
Jul 17 05:39:57.503043 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.503004    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"]
Jul 17 05:39:57.507869 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.507850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:39:57.515508 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.515017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"]
Jul 17 05:39:57.646741 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.646704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df58257b-d5ae-4226-b051-12420a0453b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k\" (UID: \"df58257b-d5ae-4226-b051-12420a0453b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:39:57.748007 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.747968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df58257b-d5ae-4226-b051-12420a0453b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k\" (UID: \"df58257b-d5ae-4226-b051-12420a0453b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:39:57.748622 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.748601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df58257b-d5ae-4226-b051-12420a0453b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k\" (UID: \"df58257b-d5ae-4226-b051-12420a0453b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:39:57.820088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.820007    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:39:57.953096 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:57.953071    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"]
Jul 17 05:39:57.955311 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:39:57.955282    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf58257b_d5ae_4226_b051_12420a0453b5.slice/crio-5d9e9ea2b76656bf6c24226c0f7288e1fcdd0e07505f82ebeb6eef6dc10765f5 WatchSource:0}: Error finding container 5d9e9ea2b76656bf6c24226c0f7288e1fcdd0e07505f82ebeb6eef6dc10765f5: Status 404 returned error can't find the container with id 5d9e9ea2b76656bf6c24226c0f7288e1fcdd0e07505f82ebeb6eef6dc10765f5
Jul 17 05:39:58.127449 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:58.127421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" event={"ID":"df58257b-d5ae-4226-b051-12420a0453b5","Type":"ContainerStarted","Data":"5d9e9ea2b76656bf6c24226c0f7288e1fcdd0e07505f82ebeb6eef6dc10765f5"}
Jul 17 05:39:58.253355 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:58.253333    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:58.354188 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:58.354154    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2873e9-55a0-40de-8cab-4265abf297cd-maas-api-service-ca\") pod \"4f2873e9-55a0-40de-8cab-4265abf297cd\" (UID: \"4f2873e9-55a0-40de-8cab-4265abf297cd\") "
Jul 17 05:39:58.354528 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:58.354506    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2873e9-55a0-40de-8cab-4265abf297cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f2873e9-55a0-40de-8cab-4265abf297cd" (UID: "4f2873e9-55a0-40de-8cab-4265abf297cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:39:58.455413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:58.455381    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2873e9-55a0-40de-8cab-4265abf297cd-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:39:59.133503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.133412    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"
Jul 17 05:39:59.133503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.133423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8" event={"ID":"4f2873e9-55a0-40de-8cab-4265abf297cd","Type":"ContainerDied","Data":"ef6adb143e52be59b8c54576f69e43af40d6dbfc8a9fa16d6447d509aece0a3a"}
Jul 17 05:39:59.133503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.133489    2582 scope.go:117] "RemoveContainer" containerID="f26c2b56ec260dd7d77a7f1ac928c6735e1aff3271122e2de9697d66fa957182"
Jul 17 05:39:59.135049 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.134993    2582 generic.go:356] "Generic (PLEG): container finished" podID="df58257b-d5ae-4226-b051-12420a0453b5" containerID="9af5aa8f5854e2b6e29d82311fe7b91e7835c08833896ab7659087b8322f91c4" exitCode=6
Jul 17 05:39:59.135049 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.135032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" event={"ID":"df58257b-d5ae-4226-b051-12420a0453b5","Type":"ContainerDied","Data":"9af5aa8f5854e2b6e29d82311fe7b91e7835c08833896ab7659087b8322f91c4"}
Jul 17 05:39:59.135300 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.135224    2582 scope.go:117] "RemoveContainer" containerID="9af5aa8f5854e2b6e29d82311fe7b91e7835c08833896ab7659087b8322f91c4"
Jul 17 05:39:59.158532 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.158503    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"]
Jul 17 05:39:59.163235 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:39:59.163212    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-hhbc8"]
Jul 17 05:40:00.142352 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:00.142316    2582 generic.go:356] "Generic (PLEG): container finished" podID="df58257b-d5ae-4226-b051-12420a0453b5" containerID="3d7762b9e89a0ed9e84faf8390b62ddc3306e534df496c3a9ba607c729c03b26" exitCode=6
Jul 17 05:40:00.143055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:00.142384    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" event={"ID":"df58257b-d5ae-4226-b051-12420a0453b5","Type":"ContainerDied","Data":"3d7762b9e89a0ed9e84faf8390b62ddc3306e534df496c3a9ba607c729c03b26"}
Jul 17 05:40:00.143055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:00.142435    2582 scope.go:117] "RemoveContainer" containerID="9af5aa8f5854e2b6e29d82311fe7b91e7835c08833896ab7659087b8322f91c4"
Jul 17 05:40:00.143055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:00.142598    2582 scope.go:117] "RemoveContainer" containerID="3d7762b9e89a0ed9e84faf8390b62ddc3306e534df496c3a9ba607c729c03b26"
Jul 17 05:40:00.143055 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:00.142798    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k_odh-ai-gateway-infra(df58257b-d5ae-4226-b051-12420a0453b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" podUID="df58257b-d5ae-4226-b051-12420a0453b5"
Jul 17 05:40:00.730087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:00.730053    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" path="/var/lib/kubelet/pods/4f2873e9-55a0-40de-8cab-4265abf297cd/volumes"
Jul 17 05:40:01.149632 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:01.149554    2582 scope.go:117] "RemoveContainer" containerID="3d7762b9e89a0ed9e84faf8390b62ddc3306e534df496c3a9ba607c729c03b26"
Jul 17 05:40:01.149994 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:01.149751    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k_odh-ai-gateway-infra(df58257b-d5ae-4226-b051-12420a0453b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" podUID="df58257b-d5ae-4226-b051-12420a0453b5"
Jul 17 05:40:09.406033 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.405995    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"]
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406312    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406323    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406343    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406349    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406398    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406406    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.406487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.406413    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:09.409195 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.409180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:09.417500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.417475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"]
Jul 17 05:40:09.448954 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.448919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8875452a-eb8e-4ef7-8c1a-264627df15ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c\" (UID: \"8875452a-eb8e-4ef7-8c1a-264627df15ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:09.549336 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.549294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8875452a-eb8e-4ef7-8c1a-264627df15ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c\" (UID: \"8875452a-eb8e-4ef7-8c1a-264627df15ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:09.549929 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.549903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8875452a-eb8e-4ef7-8c1a-264627df15ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c\" (UID: \"8875452a-eb8e-4ef7-8c1a-264627df15ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:09.720392 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.720344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:09.853823 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:09.853762    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"]
Jul 17 05:40:09.856065 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:40:09.856033    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8875452a_eb8e_4ef7_8c1a_264627df15ad.slice/crio-3b87f7eaff370fb1252e2e857e8059a2ad2ebe5675927eb79c2eb2187a93f688 WatchSource:0}: Error finding container 3b87f7eaff370fb1252e2e857e8059a2ad2ebe5675927eb79c2eb2187a93f688: Status 404 returned error can't find the container with id 3b87f7eaff370fb1252e2e857e8059a2ad2ebe5675927eb79c2eb2187a93f688
Jul 17 05:40:10.190817 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:10.190783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerStarted","Data":"e12bc59f7bbaf4f7a6ca387c666fc4f2c05aec45ace16faa3db724bc592e1e04"}
Jul 17 05:40:10.191000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:10.190836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerStarted","Data":"3b87f7eaff370fb1252e2e857e8059a2ad2ebe5675927eb79c2eb2187a93f688"}
Jul 17 05:40:10.207135 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:10.207081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" podStartSLOduration=1.207063262 podStartE2EDuration="1.207063262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:40:10.204823951 +0000 UTC m=+3013.941886291" watchObservedRunningTime="2026-07-17 05:40:10.207063262 +0000 UTC m=+3013.944125597"
Jul 17 05:40:11.197096 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:11.197066    2582 generic.go:356] "Generic (PLEG): container finished" podID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerID="e12bc59f7bbaf4f7a6ca387c666fc4f2c05aec45ace16faa3db724bc592e1e04" exitCode=6
Jul 17 05:40:11.197538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:11.197139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerDied","Data":"e12bc59f7bbaf4f7a6ca387c666fc4f2c05aec45ace16faa3db724bc592e1e04"}
Jul 17 05:40:11.197538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:11.197326    2582 scope.go:117] "RemoveContainer" containerID="e12bc59f7bbaf4f7a6ca387c666fc4f2c05aec45ace16faa3db724bc592e1e04"
Jul 17 05:40:12.203223 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:12.203185    2582 generic.go:356] "Generic (PLEG): container finished" podID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerID="1dd4e65fb820edcff083f6507fc817a60b9ceb812eca9690d45b73d5f2357e97" exitCode=6
Jul 17 05:40:12.203719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:12.203231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerDied","Data":"1dd4e65fb820edcff083f6507fc817a60b9ceb812eca9690d45b73d5f2357e97"}
Jul 17 05:40:12.203719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:12.203309    2582 scope.go:117] "RemoveContainer" containerID="e12bc59f7bbaf4f7a6ca387c666fc4f2c05aec45ace16faa3db724bc592e1e04"
Jul 17 05:40:12.203719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:12.203534    2582 scope.go:117] "RemoveContainer" containerID="1dd4e65fb820edcff083f6507fc817a60b9ceb812eca9690d45b73d5f2357e97"
Jul 17 05:40:12.203888 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:12.203783    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c_odh-ai-gateway-infra(8875452a-eb8e-4ef7-8c1a-264627df15ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad"
Jul 17 05:40:13.210129 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:13.210106    2582 scope.go:117] "RemoveContainer" containerID="1dd4e65fb820edcff083f6507fc817a60b9ceb812eca9690d45b73d5f2357e97"
Jul 17 05:40:13.210626 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:13.210306    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c_odh-ai-gateway-infra(8875452a-eb8e-4ef7-8c1a-264627df15ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad"
Jul 17 05:40:13.725215 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:13.725183    2582 scope.go:117] "RemoveContainer" containerID="3d7762b9e89a0ed9e84faf8390b62ddc3306e534df496c3a9ba607c729c03b26"
Jul 17 05:40:14.216424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:14.216386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" event={"ID":"df58257b-d5ae-4226-b051-12420a0453b5","Type":"ContainerStarted","Data":"2786ba38d3ef8c264b7a7db7aa004c2f9536956a6bb4a04294359fc06c1ed5f2"}
Jul 17 05:40:14.232837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:14.232780    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" podStartSLOduration=17.232764613 podStartE2EDuration="17.232764613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:40:14.23086426 +0000 UTC m=+3017.967926598" watchObservedRunningTime="2026-07-17 05:40:14.232764613 +0000 UTC m=+3017.969826948"
Jul 17 05:40:15.223681 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:15.223650    2582 generic.go:356] "Generic (PLEG): container finished" podID="df58257b-d5ae-4226-b051-12420a0453b5" containerID="2786ba38d3ef8c264b7a7db7aa004c2f9536956a6bb4a04294359fc06c1ed5f2" exitCode=6
Jul 17 05:40:15.224133 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:15.223711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" event={"ID":"df58257b-d5ae-4226-b051-12420a0453b5","Type":"ContainerDied","Data":"2786ba38d3ef8c264b7a7db7aa004c2f9536956a6bb4a04294359fc06c1ed5f2"}
Jul 17 05:40:15.224133 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:15.223759    2582 scope.go:117] "RemoveContainer" containerID="3d7762b9e89a0ed9e84faf8390b62ddc3306e534df496c3a9ba607c729c03b26"
Jul 17 05:40:15.224133 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:15.224031    2582 scope.go:117] "RemoveContainer" containerID="2786ba38d3ef8c264b7a7db7aa004c2f9536956a6bb4a04294359fc06c1ed5f2"
Jul 17 05:40:15.224307 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:15.224289    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k_odh-ai-gateway-infra(df58257b-d5ae-4226-b051-12420a0453b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" podUID="df58257b-d5ae-4226-b051-12420a0453b5"
Jul 17 05:40:15.247860 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:15.247830    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"]
Jul 17 05:40:16.360822 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:16.360798    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:40:16.410170 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:16.410135    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df58257b-d5ae-4226-b051-12420a0453b5-maas-api-service-ca\") pod \"df58257b-d5ae-4226-b051-12420a0453b5\" (UID: \"df58257b-d5ae-4226-b051-12420a0453b5\") "
Jul 17 05:40:16.410538 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:16.410516    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df58257b-d5ae-4226-b051-12420a0453b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df58257b-d5ae-4226-b051-12420a0453b5" (UID: "df58257b-d5ae-4226-b051-12420a0453b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:40:16.510916 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:16.510821    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df58257b-d5ae-4226-b051-12420a0453b5-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:40:17.235184 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:17.235154    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"
Jul 17 05:40:17.235184 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:17.235163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k" event={"ID":"df58257b-d5ae-4226-b051-12420a0453b5","Type":"ContainerDied","Data":"5d9e9ea2b76656bf6c24226c0f7288e1fcdd0e07505f82ebeb6eef6dc10765f5"}
Jul 17 05:40:17.235441 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:17.235226    2582 scope.go:117] "RemoveContainer" containerID="2786ba38d3ef8c264b7a7db7aa004c2f9536956a6bb4a04294359fc06c1ed5f2"
Jul 17 05:40:17.251560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:17.251526    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"]
Jul 17 05:40:17.257413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:17.257387    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-7s89k"]
Jul 17 05:40:18.729731 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:18.729697    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df58257b-d5ae-4226-b051-12420a0453b5" path="/var/lib/kubelet/pods/df58257b-d5ae-4226-b051-12420a0453b5/volumes"
Jul 17 05:40:24.829118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829081    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"]
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829399    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829411    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829423    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829430    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829440    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829445    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829453    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829458    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f2873e9-55a0-40de-8cab-4265abf297cd" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829509    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.829525 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.829517    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:24.833521 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.833503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:24.839053 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.839021    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"]
Jul 17 05:40:24.874279 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.874228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c795abd-c336-49b5-8716-86f86f6ab1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5\" (UID: \"9c795abd-c336-49b5-8716-86f86f6ab1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:24.975568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.975531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c795abd-c336-49b5-8716-86f86f6ab1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5\" (UID: \"9c795abd-c336-49b5-8716-86f86f6ab1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:24.976139 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:24.976121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c795abd-c336-49b5-8716-86f86f6ab1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5\" (UID: \"9c795abd-c336-49b5-8716-86f86f6ab1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:25.145111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:25.145024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:25.278297 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:25.278251    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"]
Jul 17 05:40:25.279917 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:40:25.279888    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c795abd_c336_49b5_8716_86f86f6ab1f5.slice/crio-d3ad24f245a0de9aee327772cdceedd72997be0030f6b7571f9b61f6fa12d1a1 WatchSource:0}: Error finding container d3ad24f245a0de9aee327772cdceedd72997be0030f6b7571f9b61f6fa12d1a1: Status 404 returned error can't find the container with id d3ad24f245a0de9aee327772cdceedd72997be0030f6b7571f9b61f6fa12d1a1
Jul 17 05:40:26.282718 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:26.282688    2582 generic.go:356] "Generic (PLEG): container finished" podID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerID="2664383ee9ee576262df4624866fa40f1fab14751f303a9725c755b1d27a4b87" exitCode=6
Jul 17 05:40:26.283118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:26.282732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" event={"ID":"9c795abd-c336-49b5-8716-86f86f6ab1f5","Type":"ContainerDied","Data":"2664383ee9ee576262df4624866fa40f1fab14751f303a9725c755b1d27a4b87"}
Jul 17 05:40:26.283118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:26.282767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" event={"ID":"9c795abd-c336-49b5-8716-86f86f6ab1f5","Type":"ContainerStarted","Data":"d3ad24f245a0de9aee327772cdceedd72997be0030f6b7571f9b61f6fa12d1a1"}
Jul 17 05:40:26.283118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:26.282922    2582 scope.go:117] "RemoveContainer" containerID="2664383ee9ee576262df4624866fa40f1fab14751f303a9725c755b1d27a4b87"
Jul 17 05:40:27.289546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:27.289460    2582 generic.go:356] "Generic (PLEG): container finished" podID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerID="19b11ee199f996a050abaf12248d4d25d34729bfae74e9119078a116d9e0762d" exitCode=6
Jul 17 05:40:27.289546 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:27.289511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" event={"ID":"9c795abd-c336-49b5-8716-86f86f6ab1f5","Type":"ContainerDied","Data":"19b11ee199f996a050abaf12248d4d25d34729bfae74e9119078a116d9e0762d"}
Jul 17 05:40:27.289985 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:27.289558    2582 scope.go:117] "RemoveContainer" containerID="2664383ee9ee576262df4624866fa40f1fab14751f303a9725c755b1d27a4b87"
Jul 17 05:40:27.289985 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:27.289747    2582 scope.go:117] "RemoveContainer" containerID="19b11ee199f996a050abaf12248d4d25d34729bfae74e9119078a116d9e0762d"
Jul 17 05:40:27.290072 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:27.290016    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5_odh-ai-gateway-infra(9c795abd-c336-49b5-8716-86f86f6ab1f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5"
Jul 17 05:40:27.725032 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:27.725002    2582 scope.go:117] "RemoveContainer" containerID="1dd4e65fb820edcff083f6507fc817a60b9ceb812eca9690d45b73d5f2357e97"
Jul 17 05:40:28.295916 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:28.295888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerStarted","Data":"46e117f8e0f01efb0be6b471bc651642282054534dc3a3fd6b059a3c7e30ff3c"}
Jul 17 05:40:28.297711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:28.297693    2582 scope.go:117] "RemoveContainer" containerID="19b11ee199f996a050abaf12248d4d25d34729bfae74e9119078a116d9e0762d"
Jul 17 05:40:28.297862 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:28.297846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5_odh-ai-gateway-infra(9c795abd-c336-49b5-8716-86f86f6ab1f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5"
Jul 17 05:40:29.304290 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:29.304234    2582 generic.go:356] "Generic (PLEG): container finished" podID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerID="46e117f8e0f01efb0be6b471bc651642282054534dc3a3fd6b059a3c7e30ff3c" exitCode=6
Jul 17 05:40:29.304673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:29.304309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerDied","Data":"46e117f8e0f01efb0be6b471bc651642282054534dc3a3fd6b059a3c7e30ff3c"}
Jul 17 05:40:29.304673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:29.304374    2582 scope.go:117] "RemoveContainer" containerID="1dd4e65fb820edcff083f6507fc817a60b9ceb812eca9690d45b73d5f2357e97"
Jul 17 05:40:29.304785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:29.304686    2582 scope.go:117] "RemoveContainer" containerID="46e117f8e0f01efb0be6b471bc651642282054534dc3a3fd6b059a3c7e30ff3c"
Jul 17 05:40:29.304953 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:29.304921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c_odh-ai-gateway-infra(8875452a-eb8e-4ef7-8c1a-264627df15ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad"
Jul 17 05:40:29.321139 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:29.321114    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"]
Jul 17 05:40:30.439520 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:30.439498    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:30.523835 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:30.523785    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8875452a-eb8e-4ef7-8c1a-264627df15ad-maas-api-service-ca\") pod \"8875452a-eb8e-4ef7-8c1a-264627df15ad\" (UID: \"8875452a-eb8e-4ef7-8c1a-264627df15ad\") "
Jul 17 05:40:30.524207 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:30.524179    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8875452a-eb8e-4ef7-8c1a-264627df15ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8875452a-eb8e-4ef7-8c1a-264627df15ad" (UID: "8875452a-eb8e-4ef7-8c1a-264627df15ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:40:30.624576 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:30.624485    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8875452a-eb8e-4ef7-8c1a-264627df15ad-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:40:31.277196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277158    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"]
Jul 17 05:40:31.277481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277469    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:31.277481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277481    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:31.277578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277494    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:31.277578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277500    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:31.277578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277561    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:31.277578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277569    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:31.277578 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.277579    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df58257b-d5ae-4226-b051-12420a0453b5" containerName="revoke-keys"
Jul 17 05:40:31.281565 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.281529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:31.287560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.287532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"]
Jul 17 05:40:31.316579 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.316558    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"
Jul 17 05:40:31.316766 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.316579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c" event={"ID":"8875452a-eb8e-4ef7-8c1a-264627df15ad","Type":"ContainerDied","Data":"3b87f7eaff370fb1252e2e857e8059a2ad2ebe5675927eb79c2eb2187a93f688"}
Jul 17 05:40:31.316766 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.316637    2582 scope.go:117] "RemoveContainer" containerID="46e117f8e0f01efb0be6b471bc651642282054534dc3a3fd6b059a3c7e30ff3c"
Jul 17 05:40:31.330956 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.330921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e313619-4f52-4912-a92c-85c280ab3f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-j79js\" (UID: \"6e313619-4f52-4912-a92c-85c280ab3f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:31.332958 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.332937    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"]
Jul 17 05:40:31.338030 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.338007    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-86x2c"]
Jul 17 05:40:31.431664 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.431630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e313619-4f52-4912-a92c-85c280ab3f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-j79js\" (UID: \"6e313619-4f52-4912-a92c-85c280ab3f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:31.432301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.432283    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e313619-4f52-4912-a92c-85c280ab3f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-j79js\" (UID: \"6e313619-4f52-4912-a92c-85c280ab3f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:31.595673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.595560    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:31.727562 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:31.727535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"]
Jul 17 05:40:31.730232 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:40:31.730201    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e313619_4f52_4912_a92c_85c280ab3f75.slice/crio-abff9c03fc91c3f6fe2fa42f2f35cf67f8152931d56febcde6a601286fb0845f WatchSource:0}: Error finding container abff9c03fc91c3f6fe2fa42f2f35cf67f8152931d56febcde6a601286fb0845f: Status 404 returned error can't find the container with id abff9c03fc91c3f6fe2fa42f2f35cf67f8152931d56febcde6a601286fb0845f
Jul 17 05:40:32.323204 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:32.323172    2582 generic.go:356] "Generic (PLEG): container finished" podID="6e313619-4f52-4912-a92c-85c280ab3f75" containerID="63962170dbdafe27e937c19b4e8ff3434a6a3b2c7849648ecb87abdee71dff9d" exitCode=6
Jul 17 05:40:32.323363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:32.323230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" event={"ID":"6e313619-4f52-4912-a92c-85c280ab3f75","Type":"ContainerDied","Data":"63962170dbdafe27e937c19b4e8ff3434a6a3b2c7849648ecb87abdee71dff9d"}
Jul 17 05:40:32.323363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:32.323289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" event={"ID":"6e313619-4f52-4912-a92c-85c280ab3f75","Type":"ContainerStarted","Data":"abff9c03fc91c3f6fe2fa42f2f35cf67f8152931d56febcde6a601286fb0845f"}
Jul 17 05:40:32.323495 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:32.323475    2582 scope.go:117] "RemoveContainer" containerID="63962170dbdafe27e937c19b4e8ff3434a6a3b2c7849648ecb87abdee71dff9d"
Jul 17 05:40:32.729852 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:32.729820    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" path="/var/lib/kubelet/pods/8875452a-eb8e-4ef7-8c1a-264627df15ad/volumes"
Jul 17 05:40:33.329729 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:33.329697    2582 generic.go:356] "Generic (PLEG): container finished" podID="6e313619-4f52-4912-a92c-85c280ab3f75" containerID="09250ba42310cd8b1135cebf1ab730a160be803dd37669cec694bc58cb529412" exitCode=6
Jul 17 05:40:33.329912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:33.329786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" event={"ID":"6e313619-4f52-4912-a92c-85c280ab3f75","Type":"ContainerDied","Data":"09250ba42310cd8b1135cebf1ab730a160be803dd37669cec694bc58cb529412"}
Jul 17 05:40:33.329912 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:33.329850    2582 scope.go:117] "RemoveContainer" containerID="63962170dbdafe27e937c19b4e8ff3434a6a3b2c7849648ecb87abdee71dff9d"
Jul 17 05:40:33.330109 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:33.330093    2582 scope.go:117] "RemoveContainer" containerID="09250ba42310cd8b1135cebf1ab730a160be803dd37669cec694bc58cb529412"
Jul 17 05:40:33.330370 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:33.330345    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-j79js_odh-ai-gateway-infra(6e313619-4f52-4912-a92c-85c280ab3f75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" podUID="6e313619-4f52-4912-a92c-85c280ab3f75"
Jul 17 05:40:34.336745 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:34.336712    2582 scope.go:117] "RemoveContainer" containerID="09250ba42310cd8b1135cebf1ab730a160be803dd37669cec694bc58cb529412"
Jul 17 05:40:34.337119 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:34.336887    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-j79js_odh-ai-gateway-infra(6e313619-4f52-4912-a92c-85c280ab3f75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" podUID="6e313619-4f52-4912-a92c-85c280ab3f75"
Jul 17 05:40:40.730643 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:40.730613    2582 scope.go:117] "RemoveContainer" containerID="19b11ee199f996a050abaf12248d4d25d34729bfae74e9119078a116d9e0762d"
Jul 17 05:40:41.370528 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:41.370493    2582 generic.go:356] "Generic (PLEG): container finished" podID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerID="40a35a8248c58751552e7d1c35c0ca33a138c65ae2ad986f7850e277f62bc6be" exitCode=6
Jul 17 05:40:41.370700 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:41.370554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" event={"ID":"9c795abd-c336-49b5-8716-86f86f6ab1f5","Type":"ContainerDied","Data":"40a35a8248c58751552e7d1c35c0ca33a138c65ae2ad986f7850e277f62bc6be"}
Jul 17 05:40:41.370700 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:41.370607    2582 scope.go:117] "RemoveContainer" containerID="19b11ee199f996a050abaf12248d4d25d34729bfae74e9119078a116d9e0762d"
Jul 17 05:40:41.370879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:41.370860    2582 scope.go:117] "RemoveContainer" containerID="40a35a8248c58751552e7d1c35c0ca33a138c65ae2ad986f7850e277f62bc6be"
Jul 17 05:40:41.371122 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:41.371096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5_odh-ai-gateway-infra(9c795abd-c336-49b5-8716-86f86f6ab1f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5"
Jul 17 05:40:42.397978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:42.397944    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"]
Jul 17 05:40:42.532418 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:42.532385    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:42.627987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:42.627948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c795abd-c336-49b5-8716-86f86f6ab1f5-maas-api-service-ca\") pod \"9c795abd-c336-49b5-8716-86f86f6ab1f5\" (UID: \"9c795abd-c336-49b5-8716-86f86f6ab1f5\") "
Jul 17 05:40:42.628358 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:42.628334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c795abd-c336-49b5-8716-86f86f6ab1f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c795abd-c336-49b5-8716-86f86f6ab1f5" (UID: "9c795abd-c336-49b5-8716-86f86f6ab1f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:40:42.728810 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:42.728778    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c795abd-c336-49b5-8716-86f86f6ab1f5-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:40:43.382639 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:43.382610    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"
Jul 17 05:40:43.382794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:43.382666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5" event={"ID":"9c795abd-c336-49b5-8716-86f86f6ab1f5","Type":"ContainerDied","Data":"d3ad24f245a0de9aee327772cdceedd72997be0030f6b7571f9b61f6fa12d1a1"}
Jul 17 05:40:43.382794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:43.382728    2582 scope.go:117] "RemoveContainer" containerID="40a35a8248c58751552e7d1c35c0ca33a138c65ae2ad986f7850e277f62bc6be"
Jul 17 05:40:43.397831 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:43.397796    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"]
Jul 17 05:40:43.400151 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:43.400129    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-trst5"]
Jul 17 05:40:44.729970 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:44.729933    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" path="/var/lib/kubelet/pods/9c795abd-c336-49b5-8716-86f86f6ab1f5/volumes"
Jul 17 05:40:47.547612 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547532    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"]
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547836    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547846    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547863    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547869    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547878    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547884    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547934    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547944    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:40:47.547982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.547951    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8875452a-eb8e-4ef7-8c1a-264627df15ad" containerName="revoke-keys"
Jul 17 05:40:47.552128 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.552111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:40:47.557753 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.557723    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"]
Jul 17 05:40:47.671712 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.671665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e4a16d-824e-4abf-9e36-c4ee33e1f4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn\" (UID: \"91e4a16d-824e-4abf-9e36-c4ee33e1f4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:40:47.725044 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.725020    2582 scope.go:117] "RemoveContainer" containerID="09250ba42310cd8b1135cebf1ab730a160be803dd37669cec694bc58cb529412"
Jul 17 05:40:47.772200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.772172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e4a16d-824e-4abf-9e36-c4ee33e1f4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn\" (UID: \"91e4a16d-824e-4abf-9e36-c4ee33e1f4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:40:47.772829 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.772810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e4a16d-824e-4abf-9e36-c4ee33e1f4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn\" (UID: \"91e4a16d-824e-4abf-9e36-c4ee33e1f4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:40:47.864321 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.864235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:40:47.993542 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:47.993407    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"]
Jul 17 05:40:47.996248 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:40:47.996220    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e4a16d_824e_4abf_9e36_c4ee33e1f4da.slice/crio-70da9098ed635d3b9b710d6864776962338623d0f091aed7109bcc9d105a988a WatchSource:0}: Error finding container 70da9098ed635d3b9b710d6864776962338623d0f091aed7109bcc9d105a988a: Status 404 returned error can't find the container with id 70da9098ed635d3b9b710d6864776962338623d0f091aed7109bcc9d105a988a
Jul 17 05:40:48.410932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.410895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" event={"ID":"91e4a16d-824e-4abf-9e36-c4ee33e1f4da","Type":"ContainerStarted","Data":"8c1b686d2aeb89b381df290925769eebf60725ec95071264fb2ef6b32e7388c9"}
Jul 17 05:40:48.411138 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.410991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" event={"ID":"91e4a16d-824e-4abf-9e36-c4ee33e1f4da","Type":"ContainerStarted","Data":"70da9098ed635d3b9b710d6864776962338623d0f091aed7109bcc9d105a988a"}
Jul 17 05:40:48.412816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.412794    2582 generic.go:356] "Generic (PLEG): container finished" podID="6e313619-4f52-4912-a92c-85c280ab3f75" containerID="07f1e8b37c76ca4f2e013b1f76a5556b298224e0ff29265bdc7dfec1b9e8eb64" exitCode=6
Jul 17 05:40:48.412949 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.412822    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" event={"ID":"6e313619-4f52-4912-a92c-85c280ab3f75","Type":"ContainerDied","Data":"07f1e8b37c76ca4f2e013b1f76a5556b298224e0ff29265bdc7dfec1b9e8eb64"}
Jul 17 05:40:48.412949 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.412859    2582 scope.go:117] "RemoveContainer" containerID="09250ba42310cd8b1135cebf1ab730a160be803dd37669cec694bc58cb529412"
Jul 17 05:40:48.413154 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.413137    2582 scope.go:117] "RemoveContainer" containerID="07f1e8b37c76ca4f2e013b1f76a5556b298224e0ff29265bdc7dfec1b9e8eb64"
Jul 17 05:40:48.413385 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:48.413362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-j79js_odh-ai-gateway-infra(6e313619-4f52-4912-a92c-85c280ab3f75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" podUID="6e313619-4f52-4912-a92c-85c280ab3f75"
Jul 17 05:40:48.427001 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:48.426960    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" podStartSLOduration=1.426946902 podStartE2EDuration="1.426946902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:40:48.425360053 +0000 UTC m=+3052.162422390" watchObservedRunningTime="2026-07-17 05:40:48.426946902 +0000 UTC m=+3052.164009238"
Jul 17 05:40:49.418894 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.418862    2582 generic.go:356] "Generic (PLEG): container finished" podID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerID="8c1b686d2aeb89b381df290925769eebf60725ec95071264fb2ef6b32e7388c9" exitCode=6
Jul 17 05:40:49.419371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.418949    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" event={"ID":"91e4a16d-824e-4abf-9e36-c4ee33e1f4da","Type":"ContainerDied","Data":"8c1b686d2aeb89b381df290925769eebf60725ec95071264fb2ef6b32e7388c9"}
Jul 17 05:40:49.419371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.419158    2582 scope.go:117] "RemoveContainer" containerID="8c1b686d2aeb89b381df290925769eebf60725ec95071264fb2ef6b32e7388c9"
Jul 17 05:40:49.449692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.449661    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"]
Jul 17 05:40:49.580747 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.580718    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:49.691246 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.691214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e313619-4f52-4912-a92c-85c280ab3f75-maas-api-service-ca\") pod \"6e313619-4f52-4912-a92c-85c280ab3f75\" (UID: \"6e313619-4f52-4912-a92c-85c280ab3f75\") "
Jul 17 05:40:49.691585 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.691563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e313619-4f52-4912-a92c-85c280ab3f75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e313619-4f52-4912-a92c-85c280ab3f75" (UID: "6e313619-4f52-4912-a92c-85c280ab3f75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:40:49.791946 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:49.791908    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e313619-4f52-4912-a92c-85c280ab3f75-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:40:50.426694 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.426602    2582 generic.go:356] "Generic (PLEG): container finished" podID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerID="fad86afd91213180df9f10d530def4381467a2b6ecc01c6405805cc9b2fa4c31" exitCode=6
Jul 17 05:40:50.426694 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.426674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" event={"ID":"91e4a16d-824e-4abf-9e36-c4ee33e1f4da","Type":"ContainerDied","Data":"fad86afd91213180df9f10d530def4381467a2b6ecc01c6405805cc9b2fa4c31"}
Jul 17 05:40:50.427199 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.426735    2582 scope.go:117] "RemoveContainer" containerID="8c1b686d2aeb89b381df290925769eebf60725ec95071264fb2ef6b32e7388c9"
Jul 17 05:40:50.427199 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.426917    2582 scope.go:117] "RemoveContainer" containerID="fad86afd91213180df9f10d530def4381467a2b6ecc01c6405805cc9b2fa4c31"
Jul 17 05:40:50.427199 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:50.427132    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn_odh-ai-gateway-infra(91e4a16d-824e-4abf-9e36-c4ee33e1f4da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da"
Jul 17 05:40:50.428008 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.427965    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"
Jul 17 05:40:50.428060 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.428011    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js" event={"ID":"6e313619-4f52-4912-a92c-85c280ab3f75","Type":"ContainerDied","Data":"abff9c03fc91c3f6fe2fa42f2f35cf67f8152931d56febcde6a601286fb0845f"}
Jul 17 05:40:50.438689 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.438668    2582 scope.go:117] "RemoveContainer" containerID="07f1e8b37c76ca4f2e013b1f76a5556b298224e0ff29265bdc7dfec1b9e8eb64"
Jul 17 05:40:50.455548 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.455518    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"]
Jul 17 05:40:50.461547 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.461524    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-j79js"]
Jul 17 05:40:50.731386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:50.731355    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" path="/var/lib/kubelet/pods/6e313619-4f52-4912-a92c-85c280ab3f75/volumes"
Jul 17 05:40:51.436561 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:40:51.436529    2582 scope.go:117] "RemoveContainer" containerID="fad86afd91213180df9f10d530def4381467a2b6ecc01c6405805cc9b2fa4c31"
Jul 17 05:40:51.437028 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:40:51.436755    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn_odh-ai-gateway-infra(91e4a16d-824e-4abf-9e36-c4ee33e1f4da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da"
Jul 17 05:41:03.724711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:03.724679    2582 scope.go:117] "RemoveContainer" containerID="fad86afd91213180df9f10d530def4381467a2b6ecc01c6405805cc9b2fa4c31"
Jul 17 05:41:04.496616 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:04.496581    2582 generic.go:356] "Generic (PLEG): container finished" podID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerID="233579964dfc63c40afdb178d890fce7897ca36846194c52c6f57857ea301332" exitCode=6
Jul 17 05:41:04.496802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:04.496657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" event={"ID":"91e4a16d-824e-4abf-9e36-c4ee33e1f4da","Type":"ContainerDied","Data":"233579964dfc63c40afdb178d890fce7897ca36846194c52c6f57857ea301332"}
Jul 17 05:41:04.496802 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:04.496718    2582 scope.go:117] "RemoveContainer" containerID="fad86afd91213180df9f10d530def4381467a2b6ecc01c6405805cc9b2fa4c31"
Jul 17 05:41:04.496995 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:04.496982    2582 scope.go:117] "RemoveContainer" containerID="233579964dfc63c40afdb178d890fce7897ca36846194c52c6f57857ea301332"
Jul 17 05:41:04.497212 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:04.497193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn_odh-ai-gateway-infra(91e4a16d-824e-4abf-9e36-c4ee33e1f4da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da"
Jul 17 05:41:05.520840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:05.520808    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"]
Jul 17 05:41:05.655745 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:05.655716    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:41:05.719927 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:05.719894    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e4a16d-824e-4abf-9e36-c4ee33e1f4da-maas-api-service-ca\") pod \"91e4a16d-824e-4abf-9e36-c4ee33e1f4da\" (UID: \"91e4a16d-824e-4abf-9e36-c4ee33e1f4da\") "
Jul 17 05:41:05.720319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:05.720293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e4a16d-824e-4abf-9e36-c4ee33e1f4da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91e4a16d-824e-4abf-9e36-c4ee33e1f4da" (UID: "91e4a16d-824e-4abf-9e36-c4ee33e1f4da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:41:05.821204 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:05.821118    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e4a16d-824e-4abf-9e36-c4ee33e1f4da-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:41:06.507382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:06.507350    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"
Jul 17 05:41:06.507579 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:06.507386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn" event={"ID":"91e4a16d-824e-4abf-9e36-c4ee33e1f4da","Type":"ContainerDied","Data":"70da9098ed635d3b9b710d6864776962338623d0f091aed7109bcc9d105a988a"}
Jul 17 05:41:06.507579 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:06.507447    2582 scope.go:117] "RemoveContainer" containerID="233579964dfc63c40afdb178d890fce7897ca36846194c52c6f57857ea301332"
Jul 17 05:41:06.532831 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:06.532801    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"]
Jul 17 05:41:06.538318 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:06.538295    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-b95hn"]
Jul 17 05:41:06.729748 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:06.729712    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" path="/var/lib/kubelet/pods/91e4a16d-824e-4abf-9e36-c4ee33e1f4da/volumes"
Jul 17 05:41:09.446311 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446279    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"]
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446615    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446628    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446634    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446642    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446646    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446657    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446662    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:41:09.446665 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446675    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446725    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446732    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446738    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446744    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446749    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:09.446960 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.446755    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c795abd-c336-49b5-8716-86f86f6ab1f5" containerName="revoke-keys"
Jul 17 05:41:09.450845 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.450826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:09.453204 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.453184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:41:09.456574 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.456552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"]
Jul 17 05:41:09.543388 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.543354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl\" (UID: \"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:09.644507 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.644477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl\" (UID: \"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:09.645125 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.645108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl\" (UID: \"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:09.762816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.762739    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:09.892934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:09.892903    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"]
Jul 17 05:41:09.896463 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:41:09.896438    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59517a1_f5a2_4a5c_b7d6_a2cf37d0e880.slice/crio-c6557fb2758dc04f5a7b2202c6e1c4fb7035e08802655f37d9f3c406cf3ef8af WatchSource:0}: Error finding container c6557fb2758dc04f5a7b2202c6e1c4fb7035e08802655f37d9f3c406cf3ef8af: Status 404 returned error can't find the container with id c6557fb2758dc04f5a7b2202c6e1c4fb7035e08802655f37d9f3c406cf3ef8af
Jul 17 05:41:10.528971 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:10.528934    2582 generic.go:356] "Generic (PLEG): container finished" podID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerID="9f811aaa63a3bfc08a5e60c2f3f3766b928b3e6668dd1ea3e2e11b50db044996" exitCode=6
Jul 17 05:41:10.529347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:10.528989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" event={"ID":"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880","Type":"ContainerDied","Data":"9f811aaa63a3bfc08a5e60c2f3f3766b928b3e6668dd1ea3e2e11b50db044996"}
Jul 17 05:41:10.529347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:10.529044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" event={"ID":"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880","Type":"ContainerStarted","Data":"c6557fb2758dc04f5a7b2202c6e1c4fb7035e08802655f37d9f3c406cf3ef8af"}
Jul 17 05:41:10.529347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:10.529220    2582 scope.go:117] "RemoveContainer" containerID="9f811aaa63a3bfc08a5e60c2f3f3766b928b3e6668dd1ea3e2e11b50db044996"
Jul 17 05:41:11.535392 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:11.535360    2582 generic.go:356] "Generic (PLEG): container finished" podID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerID="5fe445651285b4dcb28f207c5c4ef0caefc265867bc8c3afb2c95a1d24b1228f" exitCode=6
Jul 17 05:41:11.535873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:11.535425    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" event={"ID":"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880","Type":"ContainerDied","Data":"5fe445651285b4dcb28f207c5c4ef0caefc265867bc8c3afb2c95a1d24b1228f"}
Jul 17 05:41:11.535873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:11.535478    2582 scope.go:117] "RemoveContainer" containerID="9f811aaa63a3bfc08a5e60c2f3f3766b928b3e6668dd1ea3e2e11b50db044996"
Jul 17 05:41:11.535873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:11.535636    2582 scope.go:117] "RemoveContainer" containerID="5fe445651285b4dcb28f207c5c4ef0caefc265867bc8c3afb2c95a1d24b1228f"
Jul 17 05:41:11.535873 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:11.535834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl_odh-ai-gateway-infra(c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880"
Jul 17 05:41:12.542303 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:12.542247    2582 scope.go:117] "RemoveContainer" containerID="5fe445651285b4dcb28f207c5c4ef0caefc265867bc8c3afb2c95a1d24b1228f"
Jul 17 05:41:12.542692 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:12.542524    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl_odh-ai-gateway-infra(c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880"
Jul 17 05:41:14.868413 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.868382    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"]
Jul 17 05:41:14.868820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.868703    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:14.868820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.868714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:14.868820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.868732    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:14.868820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.868746    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e313619-4f52-4912-a92c-85c280ab3f75" containerName="revoke-keys"
Jul 17 05:41:14.868820 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.868811    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e4a16d-824e-4abf-9e36-c4ee33e1f4da" containerName="revoke-keys"
Jul 17 05:41:14.872977 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.872960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:14.878202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.878169    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"]
Jul 17 05:41:14.887948 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.887910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1348d0-9279-443b-86f8-7c8080e1fb1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w\" (UID: \"eb1348d0-9279-443b-86f8-7c8080e1fb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:14.989153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.989118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1348d0-9279-443b-86f8-7c8080e1fb1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w\" (UID: \"eb1348d0-9279-443b-86f8-7c8080e1fb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:14.989859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:14.989833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1348d0-9279-443b-86f8-7c8080e1fb1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w\" (UID: \"eb1348d0-9279-443b-86f8-7c8080e1fb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:15.186061 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:15.185986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:15.315111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:15.315081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"]
Jul 17 05:41:15.317135 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:41:15.317108    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1348d0_9279_443b_86f8_7c8080e1fb1d.slice/crio-73688fd0464f99897656d2052f4ed8781a41a40e0711c5bdc779c7a9bfa0a56a WatchSource:0}: Error finding container 73688fd0464f99897656d2052f4ed8781a41a40e0711c5bdc779c7a9bfa0a56a: Status 404 returned error can't find the container with id 73688fd0464f99897656d2052f4ed8781a41a40e0711c5bdc779c7a9bfa0a56a
Jul 17 05:41:15.558527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:15.558491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" event={"ID":"eb1348d0-9279-443b-86f8-7c8080e1fb1d","Type":"ContainerStarted","Data":"73688fd0464f99897656d2052f4ed8781a41a40e0711c5bdc779c7a9bfa0a56a"}
Jul 17 05:41:16.564398 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:16.564361    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerID="8ca078d57d97aeefdf2186f68a6171f3d3e16299d811fbc8bb3f3e80712b9208" exitCode=6
Jul 17 05:41:16.564893 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:16.564447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" event={"ID":"eb1348d0-9279-443b-86f8-7c8080e1fb1d","Type":"ContainerDied","Data":"8ca078d57d97aeefdf2186f68a6171f3d3e16299d811fbc8bb3f3e80712b9208"}
Jul 17 05:41:16.564893 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:16.564698    2582 scope.go:117] "RemoveContainer" containerID="8ca078d57d97aeefdf2186f68a6171f3d3e16299d811fbc8bb3f3e80712b9208"
Jul 17 05:41:17.570567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:17.570479    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerID="d850999f1c24aa2a843e1b579ef33a72bedbd7e5576c8a9167f37c334e1cd188" exitCode=6
Jul 17 05:41:17.570969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:17.570563    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" event={"ID":"eb1348d0-9279-443b-86f8-7c8080e1fb1d","Type":"ContainerDied","Data":"d850999f1c24aa2a843e1b579ef33a72bedbd7e5576c8a9167f37c334e1cd188"}
Jul 17 05:41:17.570969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:17.570627    2582 scope.go:117] "RemoveContainer" containerID="8ca078d57d97aeefdf2186f68a6171f3d3e16299d811fbc8bb3f3e80712b9208"
Jul 17 05:41:17.570969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:17.570837    2582 scope.go:117] "RemoveContainer" containerID="d850999f1c24aa2a843e1b579ef33a72bedbd7e5576c8a9167f37c334e1cd188"
Jul 17 05:41:17.571125 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:17.571055    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w_odh-ai-gateway-infra(eb1348d0-9279-443b-86f8-7c8080e1fb1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d"
Jul 17 05:41:18.576939 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:18.576914    2582 scope.go:117] "RemoveContainer" containerID="d850999f1c24aa2a843e1b579ef33a72bedbd7e5576c8a9167f37c334e1cd188"
Jul 17 05:41:18.577346 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:18.577106    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w_odh-ai-gateway-infra(eb1348d0-9279-443b-86f8-7c8080e1fb1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d"
Jul 17 05:41:21.319560 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.319514    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"]
Jul 17 05:41:21.324361 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.324332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:21.329988 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.329959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"]
Jul 17 05:41:21.339866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.339833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dc9859-fcb6-46db-a451-c5549b957b2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-lkpx5\" (UID: \"95dc9859-fcb6-46db-a451-c5549b957b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:21.441226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.441180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dc9859-fcb6-46db-a451-c5549b957b2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-lkpx5\" (UID: \"95dc9859-fcb6-46db-a451-c5549b957b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:21.441828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.441807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dc9859-fcb6-46db-a451-c5549b957b2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-lkpx5\" (UID: \"95dc9859-fcb6-46db-a451-c5549b957b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:21.635852 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.635767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:21.771643 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:21.771618    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"]
Jul 17 05:41:21.773732 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:41:21.773707    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dc9859_fcb6_46db_a451_c5549b957b2f.slice/crio-495a54b07084d1c0f84f969b5c8645af453470a6eb3c4ed5e38fc2ad4053695b WatchSource:0}: Error finding container 495a54b07084d1c0f84f969b5c8645af453470a6eb3c4ed5e38fc2ad4053695b: Status 404 returned error can't find the container with id 495a54b07084d1c0f84f969b5c8645af453470a6eb3c4ed5e38fc2ad4053695b
Jul 17 05:41:22.596828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:22.596794    2582 generic.go:356] "Generic (PLEG): container finished" podID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerID="f44b5141652f8a3af5bae244a93c9bd756e5b5cff570deddd0481e60f686787c" exitCode=6
Jul 17 05:41:22.597216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:22.596842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" event={"ID":"95dc9859-fcb6-46db-a451-c5549b957b2f","Type":"ContainerDied","Data":"f44b5141652f8a3af5bae244a93c9bd756e5b5cff570deddd0481e60f686787c"}
Jul 17 05:41:22.597216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:22.596880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" event={"ID":"95dc9859-fcb6-46db-a451-c5549b957b2f","Type":"ContainerStarted","Data":"495a54b07084d1c0f84f969b5c8645af453470a6eb3c4ed5e38fc2ad4053695b"}
Jul 17 05:41:22.597216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:22.597066    2582 scope.go:117] "RemoveContainer" containerID="f44b5141652f8a3af5bae244a93c9bd756e5b5cff570deddd0481e60f686787c"
Jul 17 05:41:23.603355 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:23.603314    2582 generic.go:356] "Generic (PLEG): container finished" podID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerID="75511cf456d6d7892eee7f8601d600ffa3b19923b5df06266f0e1159e62e5155" exitCode=6
Jul 17 05:41:23.603756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:23.603406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" event={"ID":"95dc9859-fcb6-46db-a451-c5549b957b2f","Type":"ContainerDied","Data":"75511cf456d6d7892eee7f8601d600ffa3b19923b5df06266f0e1159e62e5155"}
Jul 17 05:41:23.603756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:23.603490    2582 scope.go:117] "RemoveContainer" containerID="f44b5141652f8a3af5bae244a93c9bd756e5b5cff570deddd0481e60f686787c"
Jul 17 05:41:23.603756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:23.603656    2582 scope.go:117] "RemoveContainer" containerID="75511cf456d6d7892eee7f8601d600ffa3b19923b5df06266f0e1159e62e5155"
Jul 17 05:41:23.603902 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:23.603884    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-lkpx5_odh-ai-gateway-infra(95dc9859-fcb6-46db-a451-c5549b957b2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f"
Jul 17 05:41:24.609191 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:24.609163    2582 scope.go:117] "RemoveContainer" containerID="75511cf456d6d7892eee7f8601d600ffa3b19923b5df06266f0e1159e62e5155"
Jul 17 05:41:24.609622 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:24.609372    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-lkpx5_odh-ai-gateway-infra(95dc9859-fcb6-46db-a451-c5549b957b2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f"
Jul 17 05:41:26.727097 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:26.727067    2582 scope.go:117] "RemoveContainer" containerID="5fe445651285b4dcb28f207c5c4ef0caefc265867bc8c3afb2c95a1d24b1228f"
Jul 17 05:41:27.623884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:27.623786    2582 generic.go:356] "Generic (PLEG): container finished" podID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerID="2e533be5ce151cc732243a248fe7e9d31b9a53c965a62ca11dc86fea8a73e164" exitCode=6
Jul 17 05:41:27.623884 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:27.623866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" event={"ID":"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880","Type":"ContainerDied","Data":"2e533be5ce151cc732243a248fe7e9d31b9a53c965a62ca11dc86fea8a73e164"}
Jul 17 05:41:27.624113 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:27.623938    2582 scope.go:117] "RemoveContainer" containerID="5fe445651285b4dcb28f207c5c4ef0caefc265867bc8c3afb2c95a1d24b1228f"
Jul 17 05:41:27.624319 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:27.624299    2582 scope.go:117] "RemoveContainer" containerID="2e533be5ce151cc732243a248fe7e9d31b9a53c965a62ca11dc86fea8a73e164"
Jul 17 05:41:27.624550 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:27.624527    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl_odh-ai-gateway-infra(c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880"
Jul 17 05:41:28.654450 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:28.654417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"]
Jul 17 05:41:28.804084 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:28.804056    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:28.900014 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:28.899980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880-maas-api-service-ca\") pod \"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880\" (UID: \"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880\") "
Jul 17 05:41:28.900402 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:28.900383    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" (UID: "c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:41:29.000798 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:29.000764    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:41:29.637386 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:29.637348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl" event={"ID":"c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880","Type":"ContainerDied","Data":"c6557fb2758dc04f5a7b2202c6e1c4fb7035e08802655f37d9f3c406cf3ef8af"}
Jul 17 05:41:29.637559 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:29.637411    2582 scope.go:117] "RemoveContainer" containerID="2e533be5ce151cc732243a248fe7e9d31b9a53c965a62ca11dc86fea8a73e164"
Jul 17 05:41:29.637559 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:29.637364    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"
Jul 17 05:41:29.657600 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:29.657571    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"]
Jul 17 05:41:29.661086 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:29.661065    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-nqhdl"]
Jul 17 05:41:30.729707 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:30.729671    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" path="/var/lib/kubelet/pods/c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880/volumes"
Jul 17 05:41:31.725650 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:31.725614    2582 scope.go:117] "RemoveContainer" containerID="d850999f1c24aa2a843e1b579ef33a72bedbd7e5576c8a9167f37c334e1cd188"
Jul 17 05:41:32.652677 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:32.652592    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerID="fd2aa1ab144e603096d9522f9697397b65879122a7ceab71224cfa24e865a2a7" exitCode=6
Jul 17 05:41:32.652677 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:32.652664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" event={"ID":"eb1348d0-9279-443b-86f8-7c8080e1fb1d","Type":"ContainerDied","Data":"fd2aa1ab144e603096d9522f9697397b65879122a7ceab71224cfa24e865a2a7"}
Jul 17 05:41:32.653153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:32.652721    2582 scope.go:117] "RemoveContainer" containerID="d850999f1c24aa2a843e1b579ef33a72bedbd7e5576c8a9167f37c334e1cd188"
Jul 17 05:41:32.653153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:32.653026    2582 scope.go:117] "RemoveContainer" containerID="fd2aa1ab144e603096d9522f9697397b65879122a7ceab71224cfa24e865a2a7"
Jul 17 05:41:32.653310 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:32.653280    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w_odh-ai-gateway-infra(eb1348d0-9279-443b-86f8-7c8080e1fb1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d"
Jul 17 05:41:33.676187 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:33.676155    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"]
Jul 17 05:41:33.813487 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:33.813463    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:33.941294 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:33.941238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1348d0-9279-443b-86f8-7c8080e1fb1d-maas-api-service-ca\") pod \"eb1348d0-9279-443b-86f8-7c8080e1fb1d\" (UID: \"eb1348d0-9279-443b-86f8-7c8080e1fb1d\") "
Jul 17 05:41:33.941680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:33.941653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1348d0-9279-443b-86f8-7c8080e1fb1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb1348d0-9279-443b-86f8-7c8080e1fb1d" (UID: "eb1348d0-9279-443b-86f8-7c8080e1fb1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:41:34.041911 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.041874    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1348d0-9279-443b-86f8-7c8080e1fb1d-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:41:34.664785 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.664758    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"
Jul 17 05:41:34.665076 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.664732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w" event={"ID":"eb1348d0-9279-443b-86f8-7c8080e1fb1d","Type":"ContainerDied","Data":"73688fd0464f99897656d2052f4ed8781a41a40e0711c5bdc779c7a9bfa0a56a"}
Jul 17 05:41:34.665076 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.664908    2582 scope.go:117] "RemoveContainer" containerID="fd2aa1ab144e603096d9522f9697397b65879122a7ceab71224cfa24e865a2a7"
Jul 17 05:41:34.686990 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.686960    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"]
Jul 17 05:41:34.694524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.694497    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zvm8w"]
Jul 17 05:41:34.729819 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:34.729792    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" path="/var/lib/kubelet/pods/eb1348d0-9279-443b-86f8-7c8080e1fb1d/volumes"
Jul 17 05:41:37.580315 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580286    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"]
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580618    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580627    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580632    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580640    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580645    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580650    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580655    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580662    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580667    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580721    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580728    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580734    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.580758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.580751    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:41:37.583466 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.583451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:37.589982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.589890    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"]
Jul 17 05:41:37.671140 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.671097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0779ff8-258e-468c-a498-fb550a3a3451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx\" (UID: \"d0779ff8-258e-468c-a498-fb550a3a3451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:37.772198 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.772153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0779ff8-258e-468c-a498-fb550a3a3451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx\" (UID: \"d0779ff8-258e-468c-a498-fb550a3a3451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:37.772828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.772807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0779ff8-258e-468c-a498-fb550a3a3451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx\" (UID: \"d0779ff8-258e-468c-a498-fb550a3a3451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:37.894506 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:37.894406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:38.028342 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:38.028310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"]
Jul 17 05:41:38.031680 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:41:38.031656    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0779ff8_258e_468c_a498_fb550a3a3451.slice/crio-8f28266bec0d2590723bbb39e86393d4eb00febfe551c8c4d6483155c3ae418a WatchSource:0}: Error finding container 8f28266bec0d2590723bbb39e86393d4eb00febfe551c8c4d6483155c3ae418a: Status 404 returned error can't find the container with id 8f28266bec0d2590723bbb39e86393d4eb00febfe551c8c4d6483155c3ae418a
Jul 17 05:41:38.685372 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:38.685338    2582 generic.go:356] "Generic (PLEG): container finished" podID="d0779ff8-258e-468c-a498-fb550a3a3451" containerID="2a7873c9b8c70253864a7768a19c5756cac705dade31b173c607b355ff3f1fa5" exitCode=6
Jul 17 05:41:38.685754 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:38.685420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" event={"ID":"d0779ff8-258e-468c-a498-fb550a3a3451","Type":"ContainerDied","Data":"2a7873c9b8c70253864a7768a19c5756cac705dade31b173c607b355ff3f1fa5"}
Jul 17 05:41:38.685754 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:38.685471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" event={"ID":"d0779ff8-258e-468c-a498-fb550a3a3451","Type":"ContainerStarted","Data":"8f28266bec0d2590723bbb39e86393d4eb00febfe551c8c4d6483155c3ae418a"}
Jul 17 05:41:38.685754 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:38.685601    2582 scope.go:117] "RemoveContainer" containerID="2a7873c9b8c70253864a7768a19c5756cac705dade31b173c607b355ff3f1fa5"
Jul 17 05:41:39.692031 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:39.692004    2582 generic.go:356] "Generic (PLEG): container finished" podID="d0779ff8-258e-468c-a498-fb550a3a3451" containerID="112f7a767f83bfda7e116a9e55cb8337d19fc9a31843ca4c9ed9adc91e92e1a0" exitCode=6
Jul 17 05:41:39.692469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:39.692096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" event={"ID":"d0779ff8-258e-468c-a498-fb550a3a3451","Type":"ContainerDied","Data":"112f7a767f83bfda7e116a9e55cb8337d19fc9a31843ca4c9ed9adc91e92e1a0"}
Jul 17 05:41:39.692469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:39.692163    2582 scope.go:117] "RemoveContainer" containerID="2a7873c9b8c70253864a7768a19c5756cac705dade31b173c607b355ff3f1fa5"
Jul 17 05:41:39.692469 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:39.692396    2582 scope.go:117] "RemoveContainer" containerID="112f7a767f83bfda7e116a9e55cb8337d19fc9a31843ca4c9ed9adc91e92e1a0"
Jul 17 05:41:39.692592 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:39.692575    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx_odh-ai-gateway-infra(d0779ff8-258e-468c-a498-fb550a3a3451)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" podUID="d0779ff8-258e-468c-a498-fb550a3a3451"
Jul 17 05:41:39.724901 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:39.724880    2582 scope.go:117] "RemoveContainer" containerID="75511cf456d6d7892eee7f8601d600ffa3b19923b5df06266f0e1159e62e5155"
Jul 17 05:41:40.698628 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:40.698603    2582 scope.go:117] "RemoveContainer" containerID="112f7a767f83bfda7e116a9e55cb8337d19fc9a31843ca4c9ed9adc91e92e1a0"
Jul 17 05:41:40.699128 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:40.698849    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx_odh-ai-gateway-infra(d0779ff8-258e-468c-a498-fb550a3a3451)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" podUID="d0779ff8-258e-468c-a498-fb550a3a3451"
Jul 17 05:41:40.700205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:40.700177    2582 generic.go:356] "Generic (PLEG): container finished" podID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerID="e18c719606f240abcf296cd5a7f44d2dfde6537edaf459c331075c6c69b11a96" exitCode=6
Jul 17 05:41:40.700324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:40.700238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" event={"ID":"95dc9859-fcb6-46db-a451-c5549b957b2f","Type":"ContainerDied","Data":"e18c719606f240abcf296cd5a7f44d2dfde6537edaf459c331075c6c69b11a96"}
Jul 17 05:41:40.700324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:40.700305    2582 scope.go:117] "RemoveContainer" containerID="75511cf456d6d7892eee7f8601d600ffa3b19923b5df06266f0e1159e62e5155"
Jul 17 05:41:40.700586 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:40.700570    2582 scope.go:117] "RemoveContainer" containerID="e18c719606f240abcf296cd5a7f44d2dfde6537edaf459c331075c6c69b11a96"
Jul 17 05:41:40.700735 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:40.700715    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-lkpx5_odh-ai-gateway-infra(95dc9859-fcb6-46db-a451-c5549b957b2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f"
Jul 17 05:41:41.756952 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:41.756915    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"]
Jul 17 05:41:41.888207 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:41.888184    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:42.010732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.010641    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dc9859-fcb6-46db-a451-c5549b957b2f-maas-api-service-ca\") pod \"95dc9859-fcb6-46db-a451-c5549b957b2f\" (UID: \"95dc9859-fcb6-46db-a451-c5549b957b2f\") "
Jul 17 05:41:42.011089 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.011056    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95dc9859-fcb6-46db-a451-c5549b957b2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95dc9859-fcb6-46db-a451-c5549b957b2f" (UID: "95dc9859-fcb6-46db-a451-c5549b957b2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:41:42.112175 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.112135    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dc9859-fcb6-46db-a451-c5549b957b2f-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:41:42.712184 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.712145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5" event={"ID":"95dc9859-fcb6-46db-a451-c5549b957b2f","Type":"ContainerDied","Data":"495a54b07084d1c0f84f969b5c8645af453470a6eb3c4ed5e38fc2ad4053695b"}
Jul 17 05:41:42.712394 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.712203    2582 scope.go:117] "RemoveContainer" containerID="e18c719606f240abcf296cd5a7f44d2dfde6537edaf459c331075c6c69b11a96"
Jul 17 05:41:42.712394 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.712161    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"
Jul 17 05:41:42.732020 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.731984    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"]
Jul 17 05:41:42.736346 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:42.736317    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-lkpx5"]
Jul 17 05:41:44.730174 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:44.730126    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" path="/var/lib/kubelet/pods/95dc9859-fcb6-46db-a451-c5549b957b2f/volumes"
Jul 17 05:41:54.725202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:54.725170    2582 scope.go:117] "RemoveContainer" containerID="112f7a767f83bfda7e116a9e55cb8337d19fc9a31843ca4c9ed9adc91e92e1a0"
Jul 17 05:41:55.774253 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:55.774214    2582 generic.go:356] "Generic (PLEG): container finished" podID="d0779ff8-258e-468c-a498-fb550a3a3451" containerID="4b721f719f037209149e72b0bdd26eebaaa2b742701eb0a238d025b329c6ffa8" exitCode=6
Jul 17 05:41:55.774764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:55.774287    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" event={"ID":"d0779ff8-258e-468c-a498-fb550a3a3451","Type":"ContainerDied","Data":"4b721f719f037209149e72b0bdd26eebaaa2b742701eb0a238d025b329c6ffa8"}
Jul 17 05:41:55.774764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:55.774349    2582 scope.go:117] "RemoveContainer" containerID="112f7a767f83bfda7e116a9e55cb8337d19fc9a31843ca4c9ed9adc91e92e1a0"
Jul 17 05:41:55.774764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:55.774660    2582 scope.go:117] "RemoveContainer" containerID="4b721f719f037209149e72b0bdd26eebaaa2b742701eb0a238d025b329c6ffa8"
Jul 17 05:41:55.774933 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:41:55.774918    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx_odh-ai-gateway-infra(d0779ff8-258e-468c-a498-fb550a3a3451)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" podUID="d0779ff8-258e-468c-a498-fb550a3a3451"
Jul 17 05:41:56.799866 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:56.799835    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"]
Jul 17 05:41:56.932401 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:56.932371    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:57.036415 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.036381    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0779ff8-258e-468c-a498-fb550a3a3451-maas-api-service-ca\") pod \"d0779ff8-258e-468c-a498-fb550a3a3451\" (UID: \"d0779ff8-258e-468c-a498-fb550a3a3451\") "
Jul 17 05:41:57.036779 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.036752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0779ff8-258e-468c-a498-fb550a3a3451-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0779ff8-258e-468c-a498-fb550a3a3451" (UID: "d0779ff8-258e-468c-a498-fb550a3a3451"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:41:57.137494 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.137412    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0779ff8-258e-468c-a498-fb550a3a3451-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:41:57.786324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.786295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx" event={"ID":"d0779ff8-258e-468c-a498-fb550a3a3451","Type":"ContainerDied","Data":"8f28266bec0d2590723bbb39e86393d4eb00febfe551c8c4d6483155c3ae418a"}
Jul 17 05:41:57.786324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.786296    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"
Jul 17 05:41:57.786559 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.786356    2582 scope.go:117] "RemoveContainer" containerID="4b721f719f037209149e72b0bdd26eebaaa2b742701eb0a238d025b329c6ffa8"
Jul 17 05:41:57.809507 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.809479    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"]
Jul 17 05:41:57.814715 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:57.814693    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-47fvx"]
Jul 17 05:41:58.729695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:41:58.729659    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" path="/var/lib/kubelet/pods/d0779ff8-258e-468c-a498-fb550a3a3451/volumes"
Jul 17 05:42:04.903616 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903586    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"]
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903896    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903914    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903930    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903937    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903947    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903952    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903962    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903966    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903978    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:04.903993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.903983    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:04.904324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.904031    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59517a1-f5a2-4a5c-b7d6-a2cf37d0e880" containerName="revoke-keys"
Jul 17 05:42:04.904324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.904038    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:04.904324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.904045    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:04.904324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.904053    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1348d0-9279-443b-86f8-7c8080e1fb1d" containerName="revoke-keys"
Jul 17 05:42:04.904324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.904058    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:04.904324 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.904062    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:04.906975 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.906958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:04.909718 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.909694    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:42:04.914428 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:04.914404    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"]
Jul 17 05:42:05.009032 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.008989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09033a16-9b0d-4d44-a4f0-4612105932fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9\" (UID: \"09033a16-9b0d-4d44-a4f0-4612105932fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:05.110135 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.110101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09033a16-9b0d-4d44-a4f0-4612105932fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9\" (UID: \"09033a16-9b0d-4d44-a4f0-4612105932fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:05.110722 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.110702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09033a16-9b0d-4d44-a4f0-4612105932fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9\" (UID: \"09033a16-9b0d-4d44-a4f0-4612105932fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:05.217729 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.217699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:05.347941 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.347820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"]
Jul 17 05:42:05.350299 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:42:05.350252    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09033a16_9b0d_4d44_a4f0_4612105932fe.slice/crio-b2ca832cd11ba7b4e0aaa79a981805a58effa0fa395aa2817f485e693261dae2 WatchSource:0}: Error finding container b2ca832cd11ba7b4e0aaa79a981805a58effa0fa395aa2817f485e693261dae2: Status 404 returned error can't find the container with id b2ca832cd11ba7b4e0aaa79a981805a58effa0fa395aa2817f485e693261dae2
Jul 17 05:42:05.822631 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.822599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" event={"ID":"09033a16-9b0d-4d44-a4f0-4612105932fe","Type":"ContainerStarted","Data":"efc01c8d6eba6e08dde3c7677a8d5b8da39322407fc62084b785509f3c5b0adc"}
Jul 17 05:42:05.823095 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.822651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" event={"ID":"09033a16-9b0d-4d44-a4f0-4612105932fe","Type":"ContainerStarted","Data":"b2ca832cd11ba7b4e0aaa79a981805a58effa0fa395aa2817f485e693261dae2"}
Jul 17 05:42:05.839810 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:05.839763    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" podStartSLOduration=1.839747016 podStartE2EDuration="1.839747016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:05.838450741 +0000 UTC m=+3129.575513114" watchObservedRunningTime="2026-07-17 05:42:05.839747016 +0000 UTC m=+3129.576809352"
Jul 17 05:42:06.829793 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:06.829752    2582 generic.go:356] "Generic (PLEG): container finished" podID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerID="efc01c8d6eba6e08dde3c7677a8d5b8da39322407fc62084b785509f3c5b0adc" exitCode=6
Jul 17 05:42:06.830371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:06.829841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" event={"ID":"09033a16-9b0d-4d44-a4f0-4612105932fe","Type":"ContainerDied","Data":"efc01c8d6eba6e08dde3c7677a8d5b8da39322407fc62084b785509f3c5b0adc"}
Jul 17 05:42:06.830371 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:06.830084    2582 scope.go:117] "RemoveContainer" containerID="efc01c8d6eba6e08dde3c7677a8d5b8da39322407fc62084b785509f3c5b0adc"
Jul 17 05:42:07.835842 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:07.835812    2582 generic.go:356] "Generic (PLEG): container finished" podID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerID="aa230bb6e57557a34be8d904b86a02aa838808306e1f24d7b14de52a2bc2ec1f" exitCode=6
Jul 17 05:42:07.836301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:07.835896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" event={"ID":"09033a16-9b0d-4d44-a4f0-4612105932fe","Type":"ContainerDied","Data":"aa230bb6e57557a34be8d904b86a02aa838808306e1f24d7b14de52a2bc2ec1f"}
Jul 17 05:42:07.836301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:07.835961    2582 scope.go:117] "RemoveContainer" containerID="efc01c8d6eba6e08dde3c7677a8d5b8da39322407fc62084b785509f3c5b0adc"
Jul 17 05:42:07.836301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:07.836152    2582 scope.go:117] "RemoveContainer" containerID="aa230bb6e57557a34be8d904b86a02aa838808306e1f24d7b14de52a2bc2ec1f"
Jul 17 05:42:07.836456 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:07.836404    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9_odh-ai-gateway-infra(09033a16-9b0d-4d44-a4f0-4612105932fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe"
Jul 17 05:42:08.841457 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:08.841430    2582 scope.go:117] "RemoveContainer" containerID="aa230bb6e57557a34be8d904b86a02aa838808306e1f24d7b14de52a2bc2ec1f"
Jul 17 05:42:08.841842 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:08.841609    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9_odh-ai-gateway-infra(09033a16-9b0d-4d44-a4f0-4612105932fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe"
Jul 17 05:42:09.490834 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.490806    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"]
Jul 17 05:42:09.491142 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.491129    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:09.491189 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.491144    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:09.491189 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.491158    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:09.491189 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.491167    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:09.491320 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.491231    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0779ff8-258e-468c-a498-fb550a3a3451" containerName="revoke-keys"
Jul 17 05:42:09.491320 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.491242    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95dc9859-fcb6-46db-a451-c5549b957b2f" containerName="revoke-keys"
Jul 17 05:42:09.495465 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.495437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:09.500645 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.500614    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"]
Jul 17 05:42:09.646976 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.646928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v\" (UID: \"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:09.748501 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.748412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v\" (UID: \"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:09.749040 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.749018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v\" (UID: \"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:09.807530 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.807494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:09.941034 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:09.941009    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"]
Jul 17 05:42:09.943537 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:42:09.943507    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787b6d8f_d44b_4622_b231_4d4e6e1a2cb4.slice/crio-0a25fb7719c8a6b506fe9e585037220c8807dee5d7d1abf83131e7f8c8a5756b WatchSource:0}: Error finding container 0a25fb7719c8a6b506fe9e585037220c8807dee5d7d1abf83131e7f8c8a5756b: Status 404 returned error can't find the container with id 0a25fb7719c8a6b506fe9e585037220c8807dee5d7d1abf83131e7f8c8a5756b
Jul 17 05:42:10.851778 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:10.851691    2582 generic.go:356] "Generic (PLEG): container finished" podID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerID="0214c8931310ef63fa78ac6e9197f7ebbe83629d48c97974293516e47bae4544" exitCode=6
Jul 17 05:42:10.851956 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:10.851779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" event={"ID":"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4","Type":"ContainerDied","Data":"0214c8931310ef63fa78ac6e9197f7ebbe83629d48c97974293516e47bae4544"}
Jul 17 05:42:10.851956 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:10.851832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" event={"ID":"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4","Type":"ContainerStarted","Data":"0a25fb7719c8a6b506fe9e585037220c8807dee5d7d1abf83131e7f8c8a5756b"}
Jul 17 05:42:10.851956 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:10.851933    2582 scope.go:117] "RemoveContainer" containerID="0214c8931310ef63fa78ac6e9197f7ebbe83629d48c97974293516e47bae4544"
Jul 17 05:42:11.857542 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:11.857504    2582 generic.go:356] "Generic (PLEG): container finished" podID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerID="6130611382e4f3e2a055cfa9d3a621c26a544344088f4796221111375b8ca2d2" exitCode=6
Jul 17 05:42:11.858041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:11.857600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" event={"ID":"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4","Type":"ContainerDied","Data":"6130611382e4f3e2a055cfa9d3a621c26a544344088f4796221111375b8ca2d2"}
Jul 17 05:42:11.858041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:11.857655    2582 scope.go:117] "RemoveContainer" containerID="0214c8931310ef63fa78ac6e9197f7ebbe83629d48c97974293516e47bae4544"
Jul 17 05:42:11.858041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:11.857792    2582 scope.go:117] "RemoveContainer" containerID="6130611382e4f3e2a055cfa9d3a621c26a544344088f4796221111375b8ca2d2"
Jul 17 05:42:11.858041 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:11.858018    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v_odh-ai-gateway-infra(787b6d8f-d44b-4622-b231-4d4e6e1a2cb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4"
Jul 17 05:42:12.864153 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:12.864122    2582 scope.go:117] "RemoveContainer" containerID="6130611382e4f3e2a055cfa9d3a621c26a544344088f4796221111375b8ca2d2"
Jul 17 05:42:12.864591 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:12.864364    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v_odh-ai-gateway-infra(787b6d8f-d44b-4622-b231-4d4e6e1a2cb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4"
Jul 17 05:42:20.725617 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:20.725540    2582 scope.go:117] "RemoveContainer" containerID="aa230bb6e57557a34be8d904b86a02aa838808306e1f24d7b14de52a2bc2ec1f"
Jul 17 05:42:21.362700 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.362663    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"]
Jul 17 05:42:21.365943 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.365926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:21.374119 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.374091    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"]
Jul 17 05:42:21.445876 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.445835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5bb4b-c609-4cf7-8bcf-765c04c43218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-b9m6x\" (UID: \"8da5bb4b-c609-4cf7-8bcf-765c04c43218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:21.547182 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.547084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5bb4b-c609-4cf7-8bcf-765c04c43218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-b9m6x\" (UID: \"8da5bb4b-c609-4cf7-8bcf-765c04c43218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:21.547719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.547700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5bb4b-c609-4cf7-8bcf-765c04c43218-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-b9m6x\" (UID: \"8da5bb4b-c609-4cf7-8bcf-765c04c43218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:21.677824 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.677790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:21.808500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.808415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"]
Jul 17 05:42:21.811010 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:42:21.810975    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da5bb4b_c609_4cf7_8bcf_765c04c43218.slice/crio-955337ca600ab5dc73901f65c2464b25ceb65a3150c26dc4c318c267daae9edc WatchSource:0}: Error finding container 955337ca600ab5dc73901f65c2464b25ceb65a3150c26dc4c318c267daae9edc: Status 404 returned error can't find the container with id 955337ca600ab5dc73901f65c2464b25ceb65a3150c26dc4c318c267daae9edc
Jul 17 05:42:21.905694 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.905667    2582 generic.go:356] "Generic (PLEG): container finished" podID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerID="89c40af0360e9213581c4ca07247a94e6458ce3d9e7479573a2b909ef5480b92" exitCode=6
Jul 17 05:42:21.905895 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.905742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" event={"ID":"09033a16-9b0d-4d44-a4f0-4612105932fe","Type":"ContainerDied","Data":"89c40af0360e9213581c4ca07247a94e6458ce3d9e7479573a2b909ef5480b92"}
Jul 17 05:42:21.905895 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.905811    2582 scope.go:117] "RemoveContainer" containerID="aa230bb6e57557a34be8d904b86a02aa838808306e1f24d7b14de52a2bc2ec1f"
Jul 17 05:42:21.906175 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.906150    2582 scope.go:117] "RemoveContainer" containerID="89c40af0360e9213581c4ca07247a94e6458ce3d9e7479573a2b909ef5480b92"
Jul 17 05:42:21.906441 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:21.906419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9_odh-ai-gateway-infra(09033a16-9b0d-4d44-a4f0-4612105932fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe"
Jul 17 05:42:21.907112 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:21.907091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" event={"ID":"8da5bb4b-c609-4cf7-8bcf-765c04c43218","Type":"ContainerStarted","Data":"955337ca600ab5dc73901f65c2464b25ceb65a3150c26dc4c318c267daae9edc"}
Jul 17 05:42:22.636523 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.636492    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"]
Jul 17 05:42:22.639635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.639618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.643555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.643537    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-560f5b\"/\"kube-root-ca.crt\""
Jul 17 05:42:22.643688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.643577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-560f5b\"/\"default-dockercfg-zqffw\""
Jul 17 05:42:22.643688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.643611    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-560f5b\"/\"openshift-service-ca.crt\""
Jul 17 05:42:22.643688 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.643622    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-560f5b\"/\"sub-test-model-560f5b-kserve-self-signed-certs\""
Jul 17 05:42:22.649305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.649280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"]
Jul 17 05:42:22.656787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.656761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-dshm\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.656900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.656795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b24gv\" (UniqueName: \"kubernetes.io/projected/10d30645-1e46-46fa-9135-da4f540047e3-kube-api-access-b24gv\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.656900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.656835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-home\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.656900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.656855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-model-cache\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.656900 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.656887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d30645-1e46-46fa-9135-da4f540047e3-tls-certs\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.657039 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.656969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-tmp-dir\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.757847 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.757760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-dshm\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.757847 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.757797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b24gv\" (UniqueName: \"kubernetes.io/projected/10d30645-1e46-46fa-9135-da4f540047e3-kube-api-access-b24gv\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.757847 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.757825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-home\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.757847 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.757840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-model-cache\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.758202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.757871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d30645-1e46-46fa-9135-da4f540047e3-tls-certs\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.758202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.757895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-tmp-dir\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.758363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.758211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-tmp-dir\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.758363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.758315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-model-cache\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.758363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.758345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-home\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.760214 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.760195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-dshm\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.760468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.760453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d30645-1e46-46fa-9135-da4f540047e3-tls-certs\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.766809 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.766783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b24gv\" (UniqueName: \"kubernetes.io/projected/10d30645-1e46-46fa-9135-da4f540047e3-kube-api-access-b24gv\") pod \"sub-test-model-560f5b-kserve-564c689f47-fdbtj\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") " pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:22.915267 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.915235    2582 generic.go:356] "Generic (PLEG): container finished" podID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerID="3899fc5e1958045275dd5cb07ce7d2b563f1da40346468bcf46359981fc32d4e" exitCode=6
Jul 17 05:42:22.915690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.915316    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" event={"ID":"8da5bb4b-c609-4cf7-8bcf-765c04c43218","Type":"ContainerDied","Data":"3899fc5e1958045275dd5cb07ce7d2b563f1da40346468bcf46359981fc32d4e"}
Jul 17 05:42:22.915690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.915554    2582 scope.go:117] "RemoveContainer" containerID="3899fc5e1958045275dd5cb07ce7d2b563f1da40346468bcf46359981fc32d4e"
Jul 17 05:42:22.930115 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.930089    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"]
Jul 17 05:42:22.949212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:22.949180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:23.079055 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.079026    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:23.101012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.100982    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"]
Jul 17 05:42:23.101309 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:42:23.101289    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10d30645_1e46_46fa_9135_da4f540047e3.slice/crio-a0dc53dfbc85f9ef16512e8cd2b0b699ed6d4dfb9a7a7a26fa8767e490048fcb WatchSource:0}: Error finding container a0dc53dfbc85f9ef16512e8cd2b0b699ed6d4dfb9a7a7a26fa8767e490048fcb: Status 404 returned error can't find the container with id a0dc53dfbc85f9ef16512e8cd2b0b699ed6d4dfb9a7a7a26fa8767e490048fcb
Jul 17 05:42:23.160879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.160851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09033a16-9b0d-4d44-a4f0-4612105932fe-maas-api-service-ca\") pod \"09033a16-9b0d-4d44-a4f0-4612105932fe\" (UID: \"09033a16-9b0d-4d44-a4f0-4612105932fe\") "
Jul 17 05:42:23.161181 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.161161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09033a16-9b0d-4d44-a4f0-4612105932fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09033a16-9b0d-4d44-a4f0-4612105932fe" (UID: "09033a16-9b0d-4d44-a4f0-4612105932fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:42:23.262091 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.262011    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09033a16-9b0d-4d44-a4f0-4612105932fe-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:42:23.921942 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.921859    2582 generic.go:356] "Generic (PLEG): container finished" podID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerID="5aed031499445796223721183694d99dc58fb8c8ae7b2354e611e93b0902903e" exitCode=6
Jul 17 05:42:23.921942 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.921920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" event={"ID":"8da5bb4b-c609-4cf7-8bcf-765c04c43218","Type":"ContainerDied","Data":"5aed031499445796223721183694d99dc58fb8c8ae7b2354e611e93b0902903e"}
Jul 17 05:42:23.922475 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.921988    2582 scope.go:117] "RemoveContainer" containerID="3899fc5e1958045275dd5cb07ce7d2b563f1da40346468bcf46359981fc32d4e"
Jul 17 05:42:23.922475 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.922191    2582 scope.go:117] "RemoveContainer" containerID="5aed031499445796223721183694d99dc58fb8c8ae7b2354e611e93b0902903e"
Jul 17 05:42:23.922475 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:23.922447    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-b9m6x_odh-ai-gateway-infra(8da5bb4b-c609-4cf7-8bcf-765c04c43218)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218"
Jul 17 05:42:23.923351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.923328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9" event={"ID":"09033a16-9b0d-4d44-a4f0-4612105932fe","Type":"ContainerDied","Data":"b2ca832cd11ba7b4e0aaa79a981805a58effa0fa395aa2817f485e693261dae2"}
Jul 17 05:42:23.923351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.923342    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"
Jul 17 05:42:23.924978 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.924954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj" event={"ID":"10d30645-1e46-46fa-9135-da4f540047e3","Type":"ContainerStarted","Data":"06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412"}
Jul 17 05:42:23.925086 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.924987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj" event={"ID":"10d30645-1e46-46fa-9135-da4f540047e3","Type":"ContainerStarted","Data":"a0dc53dfbc85f9ef16512e8cd2b0b699ed6d4dfb9a7a7a26fa8767e490048fcb"}
Jul 17 05:42:23.937676 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.937659    2582 scope.go:117] "RemoveContainer" containerID="89c40af0360e9213581c4ca07247a94e6458ce3d9e7479573a2b909ef5480b92"
Jul 17 05:42:23.951870 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.951846    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"]
Jul 17 05:42:23.957720 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.957698    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-qvql9"]
Jul 17 05:42:23.970949 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:23.970911    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj" podStartSLOduration=1.970897718 podStartE2EDuration="1.970897718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:23.969629684 +0000 UTC m=+3147.706692021" watchObservedRunningTime="2026-07-17 05:42:23.970897718 +0000 UTC m=+3147.707960053"
Jul 17 05:42:24.725127 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:24.725099    2582 scope.go:117] "RemoveContainer" containerID="6130611382e4f3e2a055cfa9d3a621c26a544344088f4796221111375b8ca2d2"
Jul 17 05:42:24.730494 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:24.730473    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" path="/var/lib/kubelet/pods/09033a16-9b0d-4d44-a4f0-4612105932fe/volumes"
Jul 17 05:42:24.931985 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:24.931961    2582 scope.go:117] "RemoveContainer" containerID="5aed031499445796223721183694d99dc58fb8c8ae7b2354e611e93b0902903e"
Jul 17 05:42:24.932455 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:24.932179    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-b9m6x_odh-ai-gateway-infra(8da5bb4b-c609-4cf7-8bcf-765c04c43218)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218"
Jul 17 05:42:25.937939 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:25.937911    2582 generic.go:356] "Generic (PLEG): container finished" podID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerID="2d98dd1aa018b5894c0873a0c0f07d6010adfb3f3d162a694fbe752a24636ddd" exitCode=6
Jul 17 05:42:25.938410 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:25.937985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" event={"ID":"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4","Type":"ContainerDied","Data":"2d98dd1aa018b5894c0873a0c0f07d6010adfb3f3d162a694fbe752a24636ddd"}
Jul 17 05:42:25.938410 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:25.938045    2582 scope.go:117] "RemoveContainer" containerID="6130611382e4f3e2a055cfa9d3a621c26a544344088f4796221111375b8ca2d2"
Jul 17 05:42:25.938410 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:25.938333    2582 scope.go:117] "RemoveContainer" containerID="2d98dd1aa018b5894c0873a0c0f07d6010adfb3f3d162a694fbe752a24636ddd"
Jul 17 05:42:25.938566 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:25.938543    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v_odh-ai-gateway-infra(787b6d8f-d44b-4622-b231-4d4e6e1a2cb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4"
Jul 17 05:42:26.962969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:26.962937    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"]
Jul 17 05:42:27.095791 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.095767    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:27.195287 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.195243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4-maas-api-service-ca\") pod \"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4\" (UID: \"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4\") "
Jul 17 05:42:27.195637 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.195615    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" (UID: "787b6d8f-d44b-4622-b231-4d4e6e1a2cb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:42:27.296079 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.295989    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:42:27.950117 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.950087    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"
Jul 17 05:42:27.950328 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.950133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v" event={"ID":"787b6d8f-d44b-4622-b231-4d4e6e1a2cb4","Type":"ContainerDied","Data":"0a25fb7719c8a6b506fe9e585037220c8807dee5d7d1abf83131e7f8c8a5756b"}
Jul 17 05:42:27.950328 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.950188    2582 scope.go:117] "RemoveContainer" containerID="2d98dd1aa018b5894c0873a0c0f07d6010adfb3f3d162a694fbe752a24636ddd"
Jul 17 05:42:27.975359 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.975328    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"]
Jul 17 05:42:27.980875 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:27.980843    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-5ph2v"]
Jul 17 05:42:28.730384 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:28.730353    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" path="/var/lib/kubelet/pods/787b6d8f-d44b-4622-b231-4d4e6e1a2cb4/volumes"
Jul 17 05:42:32.950276 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:32.950229    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:32.950675 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:32.950315    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:32.962862 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:32.962835    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:32.985768 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:32.985740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:42:36.726753 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:36.726721    2582 scope.go:117] "RemoveContainer" containerID="5aed031499445796223721183694d99dc58fb8c8ae7b2354e611e93b0902903e"
Jul 17 05:42:38.001468 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:38.001431    2582 generic.go:356] "Generic (PLEG): container finished" podID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerID="d8de0c4c688d28f92a302650af8f9f958154f090dafa1a78a6cf555bd50cc342" exitCode=6
Jul 17 05:42:38.001937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:38.001498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" event={"ID":"8da5bb4b-c609-4cf7-8bcf-765c04c43218","Type":"ContainerDied","Data":"d8de0c4c688d28f92a302650af8f9f958154f090dafa1a78a6cf555bd50cc342"}
Jul 17 05:42:38.001937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:38.001555    2582 scope.go:117] "RemoveContainer" containerID="5aed031499445796223721183694d99dc58fb8c8ae7b2354e611e93b0902903e"
Jul 17 05:42:38.001937 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:38.001802    2582 scope.go:117] "RemoveContainer" containerID="d8de0c4c688d28f92a302650af8f9f958154f090dafa1a78a6cf555bd50cc342"
Jul 17 05:42:38.002086 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:42:38.002007    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-b9m6x_odh-ai-gateway-infra(8da5bb4b-c609-4cf7-8bcf-765c04c43218)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218"
Jul 17 05:42:39.029086 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:39.029054    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"]
Jul 17 05:42:39.160189 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:39.160161    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:39.201270 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:39.201223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5bb4b-c609-4cf7-8bcf-765c04c43218-maas-api-service-ca\") pod \"8da5bb4b-c609-4cf7-8bcf-765c04c43218\" (UID: \"8da5bb4b-c609-4cf7-8bcf-765c04c43218\") "
Jul 17 05:42:39.201625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:39.201604    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da5bb4b-c609-4cf7-8bcf-765c04c43218-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8da5bb4b-c609-4cf7-8bcf-765c04c43218" (UID: "8da5bb4b-c609-4cf7-8bcf-765c04c43218"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:42:39.302743 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:39.302651    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5bb4b-c609-4cf7-8bcf-765c04c43218-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:42:40.013351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:40.013312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x" event={"ID":"8da5bb4b-c609-4cf7-8bcf-765c04c43218","Type":"ContainerDied","Data":"955337ca600ab5dc73901f65c2464b25ceb65a3150c26dc4c318c267daae9edc"}
Jul 17 05:42:40.013351 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:40.013340    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"
Jul 17 05:42:40.013613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:40.013381    2582 scope.go:117] "RemoveContainer" containerID="d8de0c4c688d28f92a302650af8f9f958154f090dafa1a78a6cf555bd50cc342"
Jul 17 05:42:40.033612 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:40.033575    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"]
Jul 17 05:42:40.038832 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:40.038804    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-b9m6x"]
Jul 17 05:42:40.730434 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:40.730403    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" path="/var/lib/kubelet/pods/8da5bb4b-c609-4cf7-8bcf-765c04c43218/volumes"
Jul 17 05:42:59.532769 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.532728    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"]
Jul 17 05:42:59.533224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533164    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:42:59.533224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533183    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:42:59.533224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533194    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:42:59.533224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533201    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:42:59.533224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533210    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:42:59.533224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533219    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533237    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533249    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533276    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533351    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533378    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533474    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533485    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533493    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533503    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533512    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:42:59.533582 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.533521    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:42:59.538410 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.538379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:42:59.541233 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.541216    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:42:59.544474 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.544450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"]
Jul 17 05:42:59.682936 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.682906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a288f3-9f22-433f-942c-c9486cc58dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn\" (UID: \"83a288f3-9f22-433f-942c-c9486cc58dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:42:59.783577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.783486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a288f3-9f22-433f-942c-c9486cc58dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn\" (UID: \"83a288f3-9f22-433f-942c-c9486cc58dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:42:59.784092 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.784072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a288f3-9f22-433f-942c-c9486cc58dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn\" (UID: \"83a288f3-9f22-433f-942c-c9486cc58dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:42:59.851069 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.851027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:42:59.996763 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:42:59.996740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"]
Jul 17 05:42:59.998654 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:42:59.998626    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a288f3_9f22_433f_942c_c9486cc58dbc.slice/crio-3955ff9d84e395e4c11c9a42f423147916da9f277642819f95b1d968e3e4ecf2 WatchSource:0}: Error finding container 3955ff9d84e395e4c11c9a42f423147916da9f277642819f95b1d968e3e4ecf2: Status 404 returned error can't find the container with id 3955ff9d84e395e4c11c9a42f423147916da9f277642819f95b1d968e3e4ecf2
Jul 17 05:43:00.108837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:00.108742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" event={"ID":"83a288f3-9f22-433f-942c-c9486cc58dbc","Type":"ContainerStarted","Data":"3955ff9d84e395e4c11c9a42f423147916da9f277642819f95b1d968e3e4ecf2"}
Jul 17 05:43:01.114524 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:01.114495    2582 generic.go:356] "Generic (PLEG): container finished" podID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerID="6aaafa8bcf67d76b05a841e0e49317c2de82eeffb40c0f2388df743b1a8ceb3c" exitCode=6
Jul 17 05:43:01.114931 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:01.114540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" event={"ID":"83a288f3-9f22-433f-942c-c9486cc58dbc","Type":"ContainerDied","Data":"6aaafa8bcf67d76b05a841e0e49317c2de82eeffb40c0f2388df743b1a8ceb3c"}
Jul 17 05:43:01.114931 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:01.114753    2582 scope.go:117] "RemoveContainer" containerID="6aaafa8bcf67d76b05a841e0e49317c2de82eeffb40c0f2388df743b1a8ceb3c"
Jul 17 05:43:02.120757 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:02.120725    2582 generic.go:356] "Generic (PLEG): container finished" podID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerID="c9bd6be2b3b8fb003665a39b4bf274b9b156ad2ba26247c76c1b0f7704f27c5f" exitCode=6
Jul 17 05:43:02.121200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:02.120815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" event={"ID":"83a288f3-9f22-433f-942c-c9486cc58dbc","Type":"ContainerDied","Data":"c9bd6be2b3b8fb003665a39b4bf274b9b156ad2ba26247c76c1b0f7704f27c5f"}
Jul 17 05:43:02.121200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:02.120878    2582 scope.go:117] "RemoveContainer" containerID="6aaafa8bcf67d76b05a841e0e49317c2de82eeffb40c0f2388df743b1a8ceb3c"
Jul 17 05:43:02.121200 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:02.121053    2582 scope.go:117] "RemoveContainer" containerID="c9bd6be2b3b8fb003665a39b4bf274b9b156ad2ba26247c76c1b0f7704f27c5f"
Jul 17 05:43:02.121359 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:02.121316    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn_odh-ai-gateway-infra(83a288f3-9f22-433f-942c-c9486cc58dbc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc"
Jul 17 05:43:03.127231 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:03.127203    2582 scope.go:117] "RemoveContainer" containerID="c9bd6be2b3b8fb003665a39b4bf274b9b156ad2ba26247c76c1b0f7704f27c5f"
Jul 17 05:43:03.127636 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:03.127388    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn_odh-ai-gateway-infra(83a288f3-9f22-433f-942c-c9486cc58dbc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc"
Jul 17 05:43:14.725099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:14.725070    2582 scope.go:117] "RemoveContainer" containerID="c9bd6be2b3b8fb003665a39b4bf274b9b156ad2ba26247c76c1b0f7704f27c5f"
Jul 17 05:43:15.185797 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:15.185719    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" event={"ID":"83a288f3-9f22-433f-942c-c9486cc58dbc","Type":"ContainerStarted","Data":"fdea4a44d7900ff2d46dc689c35db80578d26969adf573fa0121e5e45488d028"}
Jul 17 05:43:15.201601 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:15.201538    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" podStartSLOduration=16.20151439 podStartE2EDuration="16.20151439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:43:15.200325279 +0000 UTC m=+3198.937387634" watchObservedRunningTime="2026-07-17 05:43:15.20151439 +0000 UTC m=+3198.938576727"
Jul 17 05:43:16.192398 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:16.192362    2582 generic.go:356] "Generic (PLEG): container finished" podID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerID="fdea4a44d7900ff2d46dc689c35db80578d26969adf573fa0121e5e45488d028" exitCode=6
Jul 17 05:43:16.192887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:16.192411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" event={"ID":"83a288f3-9f22-433f-942c-c9486cc58dbc","Type":"ContainerDied","Data":"fdea4a44d7900ff2d46dc689c35db80578d26969adf573fa0121e5e45488d028"}
Jul 17 05:43:16.192887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:16.192471    2582 scope.go:117] "RemoveContainer" containerID="c9bd6be2b3b8fb003665a39b4bf274b9b156ad2ba26247c76c1b0f7704f27c5f"
Jul 17 05:43:16.192887 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:16.192734    2582 scope.go:117] "RemoveContainer" containerID="fdea4a44d7900ff2d46dc689c35db80578d26969adf573fa0121e5e45488d028"
Jul 17 05:43:16.193009 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:16.192921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn_odh-ai-gateway-infra(83a288f3-9f22-433f-942c-c9486cc58dbc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc"
Jul 17 05:43:16.209000 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:16.208976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"]
Jul 17 05:43:17.323224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:17.323198    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:43:17.325198 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:17.325175    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a288f3-9f22-433f-942c-c9486cc58dbc-maas-api-service-ca\") pod \"83a288f3-9f22-433f-942c-c9486cc58dbc\" (UID: \"83a288f3-9f22-433f-942c-c9486cc58dbc\") "
Jul 17 05:43:17.325541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:17.325520    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a288f3-9f22-433f-942c-c9486cc58dbc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83a288f3-9f22-433f-942c-c9486cc58dbc" (UID: "83a288f3-9f22-433f-942c-c9486cc58dbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:43:17.426037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:17.426002    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a288f3-9f22-433f-942c-c9486cc58dbc-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:18.204432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:18.204390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn" event={"ID":"83a288f3-9f22-433f-942c-c9486cc58dbc","Type":"ContainerDied","Data":"3955ff9d84e395e4c11c9a42f423147916da9f277642819f95b1d968e3e4ecf2"}
Jul 17 05:43:18.204432 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:18.204424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"
Jul 17 05:43:18.204686 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:18.204470    2582 scope.go:117] "RemoveContainer" containerID="fdea4a44d7900ff2d46dc689c35db80578d26969adf573fa0121e5e45488d028"
Jul 17 05:43:18.226952 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:18.226920    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"]
Jul 17 05:43:18.231341 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:18.231313    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-wvsfn"]
Jul 17 05:43:18.730111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:18.730082    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" path="/var/lib/kubelet/pods/83a288f3-9f22-433f-942c-c9486cc58dbc/volumes"
Jul 17 05:43:32.006306 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:32.006246    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"]
Jul 17 05:43:47.027135 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.027036    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj" podUID="10d30645-1e46-46fa-9135-da4f540047e3" containerName="main" containerID="cri-o://06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412" gracePeriod=45
Jul 17 05:43:47.272289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.272243    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:43:47.333282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.333167    2582 generic.go:356] "Generic (PLEG): container finished" podID="10d30645-1e46-46fa-9135-da4f540047e3" containerID="06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412" exitCode=0
Jul 17 05:43:47.333282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.333207    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"
Jul 17 05:43:47.333282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.333221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj" event={"ID":"10d30645-1e46-46fa-9135-da4f540047e3","Type":"ContainerDied","Data":"06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412"}
Jul 17 05:43:47.333282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.333279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj" event={"ID":"10d30645-1e46-46fa-9135-da4f540047e3","Type":"ContainerDied","Data":"a0dc53dfbc85f9ef16512e8cd2b0b699ed6d4dfb9a7a7a26fa8767e490048fcb"}
Jul 17 05:43:47.333591 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.333303    2582 scope.go:117] "RemoveContainer" containerID="06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412"
Jul 17 05:43:47.341248 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.341228    2582 scope.go:117] "RemoveContainer" containerID="06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412"
Jul 17 05:43:47.341532 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:47.341511    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412\": container with ID starting with 06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412 not found: ID does not exist" containerID="06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412"
Jul 17 05:43:47.341605 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.341546    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412"} err="failed to get container status \"06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412\": rpc error: code = NotFound desc = could not find container \"06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412\": container with ID starting with 06ffb94e7db22268e5ec823fc2f315e0bb9545a41f5a4da0b84034b7402c6412 not found: ID does not exist"
Jul 17 05:43:47.376518 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d30645-1e46-46fa-9135-da4f540047e3-tls-certs\") pod \"10d30645-1e46-46fa-9135-da4f540047e3\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") "
Jul 17 05:43:47.376690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-model-cache\") pod \"10d30645-1e46-46fa-9135-da4f540047e3\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") "
Jul 17 05:43:47.376690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b24gv\" (UniqueName: \"kubernetes.io/projected/10d30645-1e46-46fa-9135-da4f540047e3-kube-api-access-b24gv\") pod \"10d30645-1e46-46fa-9135-da4f540047e3\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") "
Jul 17 05:43:47.376690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-tmp-dir\") pod \"10d30645-1e46-46fa-9135-da4f540047e3\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") "
Jul 17 05:43:47.376690 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-dshm\") pod \"10d30645-1e46-46fa-9135-da4f540047e3\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") "
Jul 17 05:43:47.376924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-home\") pod \"10d30645-1e46-46fa-9135-da4f540047e3\" (UID: \"10d30645-1e46-46fa-9135-da4f540047e3\") "
Jul 17 05:43:47.376924 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "10d30645-1e46-46fa-9135-da4f540047e3" (UID: "10d30645-1e46-46fa-9135-da4f540047e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:43:47.377032 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376947    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10d30645-1e46-46fa-9135-da4f540047e3" (UID: "10d30645-1e46-46fa-9135-da4f540047e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:43:47.377032 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.376979    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-home" (OuterVolumeSpecName: "home") pod "10d30645-1e46-46fa-9135-da4f540047e3" (UID: "10d30645-1e46-46fa-9135-da4f540047e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:43:47.377115 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.377035    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-model-cache\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:47.377115 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.377055    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-tmp-dir\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:47.377115 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.377066    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-home\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:47.378934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.378906    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-dshm" (OuterVolumeSpecName: "dshm") pod "10d30645-1e46-46fa-9135-da4f540047e3" (UID: "10d30645-1e46-46fa-9135-da4f540047e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:43:47.379205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.379179    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d30645-1e46-46fa-9135-da4f540047e3-kube-api-access-b24gv" (OuterVolumeSpecName: "kube-api-access-b24gv") pod "10d30645-1e46-46fa-9135-da4f540047e3" (UID: "10d30645-1e46-46fa-9135-da4f540047e3"). InnerVolumeSpecName "kube-api-access-b24gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:43:47.379322 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.379219    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d30645-1e46-46fa-9135-da4f540047e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10d30645-1e46-46fa-9135-da4f540047e3" (UID: "10d30645-1e46-46fa-9135-da4f540047e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:43:47.477503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.477471    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d30645-1e46-46fa-9135-da4f540047e3-tls-certs\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:47.477503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.477498    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b24gv\" (UniqueName: \"kubernetes.io/projected/10d30645-1e46-46fa-9135-da4f540047e3-kube-api-access-b24gv\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:47.477503 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.477512    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10d30645-1e46-46fa-9135-da4f540047e3-dshm\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:43:47.653087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.653056    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"]
Jul 17 05:43:47.658547 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:47.658518    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-560f5b/sub-test-model-560f5b-kserve-564c689f47-fdbtj"]
Jul 17 05:43:48.729579 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:48.729548    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d30645-1e46-46fa-9135-da4f540047e3" path="/var/lib/kubelet/pods/10d30645-1e46-46fa-9135-da4f540047e3/volumes"
Jul 17 05:43:49.571332 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571297    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"]
Jul 17 05:43:49.571606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571593    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571608    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571616    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571621    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571627    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571632    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571641    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571646    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:43:49.571658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571658    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d30645-1e46-46fa-9135-da4f540047e3" containerName="main"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571663    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d30645-1e46-46fa-9135-da4f540047e3" containerName="main"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571677    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571682    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571726    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571733    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571738    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571745    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da5bb4b-c609-4cf7-8bcf-765c04c43218" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571751    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="10d30645-1e46-46fa-9135-da4f540047e3" containerName="main"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571757    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09033a16-9b0d-4d44-a4f0-4612105932fe" containerName="revoke-keys"
Jul 17 05:43:49.571902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.571763    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="787b6d8f-d44b-4622-b231-4d4e6e1a2cb4" containerName="revoke-keys"
Jul 17 05:43:49.576072 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.576054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:43:49.578752 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.578728    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:43:49.580687 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.580663    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"]
Jul 17 05:43:49.696784 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.696744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711a5ee2-cc11-459a-9efa-93df536aba61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc\" (UID: \"711a5ee2-cc11-459a-9efa-93df536aba61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:43:49.797986 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.797934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711a5ee2-cc11-459a-9efa-93df536aba61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc\" (UID: \"711a5ee2-cc11-459a-9efa-93df536aba61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:43:49.798654 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.798634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711a5ee2-cc11-459a-9efa-93df536aba61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc\" (UID: \"711a5ee2-cc11-459a-9efa-93df536aba61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:43:49.888088 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:49.888003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:43:50.016961 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:50.016936    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"]
Jul 17 05:43:50.019029 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:43:50.019004    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711a5ee2_cc11_459a_9efa_93df536aba61.slice/crio-e7cb0216b6d41b92c40720041ed17dd1e0f33ab3278fb8650de406c81aaf1f2d WatchSource:0}: Error finding container e7cb0216b6d41b92c40720041ed17dd1e0f33ab3278fb8650de406c81aaf1f2d: Status 404 returned error can't find the container with id e7cb0216b6d41b92c40720041ed17dd1e0f33ab3278fb8650de406c81aaf1f2d
Jul 17 05:43:50.349338 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:50.349295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" event={"ID":"711a5ee2-cc11-459a-9efa-93df536aba61","Type":"ContainerStarted","Data":"b51e23f4329ce1884374147582d56338798e546c3801de45bb977086717e1281"}
Jul 17 05:43:50.349527 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:50.349364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" event={"ID":"711a5ee2-cc11-459a-9efa-93df536aba61","Type":"ContainerStarted","Data":"e7cb0216b6d41b92c40720041ed17dd1e0f33ab3278fb8650de406c81aaf1f2d"}
Jul 17 05:43:50.365265 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:50.365205    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" podStartSLOduration=1.365190363 podStartE2EDuration="1.365190363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:43:50.363818182 +0000 UTC m=+3234.100880518" watchObservedRunningTime="2026-07-17 05:43:50.365190363 +0000 UTC m=+3234.102252710"
Jul 17 05:43:51.355152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:51.355122    2582 generic.go:356] "Generic (PLEG): container finished" podID="711a5ee2-cc11-459a-9efa-93df536aba61" containerID="b51e23f4329ce1884374147582d56338798e546c3801de45bb977086717e1281" exitCode=6
Jul 17 05:43:51.355567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:51.355175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" event={"ID":"711a5ee2-cc11-459a-9efa-93df536aba61","Type":"ContainerDied","Data":"b51e23f4329ce1884374147582d56338798e546c3801de45bb977086717e1281"}
Jul 17 05:43:51.355567 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:51.355427    2582 scope.go:117] "RemoveContainer" containerID="b51e23f4329ce1884374147582d56338798e546c3801de45bb977086717e1281"
Jul 17 05:43:52.361653 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:52.361569    2582 generic.go:356] "Generic (PLEG): container finished" podID="711a5ee2-cc11-459a-9efa-93df536aba61" containerID="35d5fc8d0b8033318d2e62dfe0caeedf53844cd547f04e32aec45f10f45a46fb" exitCode=6
Jul 17 05:43:52.362023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:52.361663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" event={"ID":"711a5ee2-cc11-459a-9efa-93df536aba61","Type":"ContainerDied","Data":"35d5fc8d0b8033318d2e62dfe0caeedf53844cd547f04e32aec45f10f45a46fb"}
Jul 17 05:43:52.362023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:52.361725    2582 scope.go:117] "RemoveContainer" containerID="b51e23f4329ce1884374147582d56338798e546c3801de45bb977086717e1281"
Jul 17 05:43:52.362023 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:52.361907    2582 scope.go:117] "RemoveContainer" containerID="35d5fc8d0b8033318d2e62dfe0caeedf53844cd547f04e32aec45f10f45a46fb"
Jul 17 05:43:52.362166 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:52.362148    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc_odh-ai-gateway-infra(711a5ee2-cc11-459a-9efa-93df536aba61)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" podUID="711a5ee2-cc11-459a-9efa-93df536aba61"
Jul 17 05:43:53.368314 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:53.368286    2582 scope.go:117] "RemoveContainer" containerID="35d5fc8d0b8033318d2e62dfe0caeedf53844cd547f04e32aec45f10f45a46fb"
Jul 17 05:43:53.368799 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:53.368515    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc_odh-ai-gateway-infra(711a5ee2-cc11-459a-9efa-93df536aba61)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" podUID="711a5ee2-cc11-459a-9efa-93df536aba61"
Jul 17 05:43:54.987117 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:54.987078    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"]
Jul 17 05:43:54.987554 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:54.987422    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:54.987554 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:54.987434    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a288f3-9f22-433f-942c-c9486cc58dbc" containerName="revoke-keys"
Jul 17 05:43:54.993213 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:54.993196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:43:54.996913 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:54.996886    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"]
Jul 17 05:43:55.141930 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:55.141891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20bbeb9-471c-4de7-88ca-a28f3297e7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh\" (UID: \"f20bbeb9-471c-4de7-88ca-a28f3297e7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:43:55.243226 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:55.243132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20bbeb9-471c-4de7-88ca-a28f3297e7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh\" (UID: \"f20bbeb9-471c-4de7-88ca-a28f3297e7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:43:55.243856 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:55.243837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20bbeb9-471c-4de7-88ca-a28f3297e7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh\" (UID: \"f20bbeb9-471c-4de7-88ca-a28f3297e7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:43:55.305535 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:55.305481    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:43:55.438326 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:55.438296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"]
Jul 17 05:43:55.440326 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:43:55.440298    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20bbeb9_471c_4de7_88ca_a28f3297e7d4.slice/crio-fedbac69cfe22e49eb86d844e29ddc2d0bc2b5a82766267b50337e85454f1034 WatchSource:0}: Error finding container fedbac69cfe22e49eb86d844e29ddc2d0bc2b5a82766267b50337e85454f1034: Status 404 returned error can't find the container with id fedbac69cfe22e49eb86d844e29ddc2d0bc2b5a82766267b50337e85454f1034
Jul 17 05:43:56.384620 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:56.384527    2582 generic.go:356] "Generic (PLEG): container finished" podID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerID="c6ede356393fa372342065d84e573508dee443e62fb05551c2f0c295f4139e3f" exitCode=6
Jul 17 05:43:56.385012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:56.384613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" event={"ID":"f20bbeb9-471c-4de7-88ca-a28f3297e7d4","Type":"ContainerDied","Data":"c6ede356393fa372342065d84e573508dee443e62fb05551c2f0c295f4139e3f"}
Jul 17 05:43:56.385012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:56.384676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" event={"ID":"f20bbeb9-471c-4de7-88ca-a28f3297e7d4","Type":"ContainerStarted","Data":"fedbac69cfe22e49eb86d844e29ddc2d0bc2b5a82766267b50337e85454f1034"}
Jul 17 05:43:56.385012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:56.384790    2582 scope.go:117] "RemoveContainer" containerID="c6ede356393fa372342065d84e573508dee443e62fb05551c2f0c295f4139e3f"
Jul 17 05:43:57.391250 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:57.391223    2582 generic.go:356] "Generic (PLEG): container finished" podID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerID="4c2d9fdb815b55ca9510e76c11d619334a9b8b41ed9ae9e9c0d1e9aaf0a1b916" exitCode=6
Jul 17 05:43:57.391745 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:57.391300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" event={"ID":"f20bbeb9-471c-4de7-88ca-a28f3297e7d4","Type":"ContainerDied","Data":"4c2d9fdb815b55ca9510e76c11d619334a9b8b41ed9ae9e9c0d1e9aaf0a1b916"}
Jul 17 05:43:57.391745 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:57.391375    2582 scope.go:117] "RemoveContainer" containerID="c6ede356393fa372342065d84e573508dee443e62fb05551c2f0c295f4139e3f"
Jul 17 05:43:57.391745 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:57.391546    2582 scope.go:117] "RemoveContainer" containerID="4c2d9fdb815b55ca9510e76c11d619334a9b8b41ed9ae9e9c0d1e9aaf0a1b916"
Jul 17 05:43:57.391882 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:57.391760    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh_odh-ai-gateway-infra(f20bbeb9-471c-4de7-88ca-a28f3297e7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4"
Jul 17 05:43:58.403945 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:43:58.403918    2582 scope.go:117] "RemoveContainer" containerID="4c2d9fdb815b55ca9510e76c11d619334a9b8b41ed9ae9e9c0d1e9aaf0a1b916"
Jul 17 05:43:58.404363 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:43:58.404096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh_odh-ai-gateway-infra(f20bbeb9-471c-4de7-88ca-a28f3297e7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4"
Jul 17 05:44:04.725581 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:04.725547    2582 scope.go:117] "RemoveContainer" containerID="35d5fc8d0b8033318d2e62dfe0caeedf53844cd547f04e32aec45f10f45a46fb"
Jul 17 05:44:05.436957 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:05.436919    2582 generic.go:356] "Generic (PLEG): container finished" podID="711a5ee2-cc11-459a-9efa-93df536aba61" containerID="0f094298b0b330ead799acaefd029601963b40376eb908c31f120aa299770bed" exitCode=6
Jul 17 05:44:05.437114 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:05.436993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" event={"ID":"711a5ee2-cc11-459a-9efa-93df536aba61","Type":"ContainerDied","Data":"0f094298b0b330ead799acaefd029601963b40376eb908c31f120aa299770bed"}
Jul 17 05:44:05.437114 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:05.437085    2582 scope.go:117] "RemoveContainer" containerID="35d5fc8d0b8033318d2e62dfe0caeedf53844cd547f04e32aec45f10f45a46fb"
Jul 17 05:44:05.437414 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:05.437395    2582 scope.go:117] "RemoveContainer" containerID="0f094298b0b330ead799acaefd029601963b40376eb908c31f120aa299770bed"
Jul 17 05:44:05.437588 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:05.437571    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc_odh-ai-gateway-infra(711a5ee2-cc11-459a-9efa-93df536aba61)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" podUID="711a5ee2-cc11-459a-9efa-93df536aba61"
Jul 17 05:44:06.461569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:06.461538    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"]
Jul 17 05:44:06.592917 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:06.592895    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:44:06.743928 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:06.743852    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711a5ee2-cc11-459a-9efa-93df536aba61-maas-api-service-ca\") pod \"711a5ee2-cc11-459a-9efa-93df536aba61\" (UID: \"711a5ee2-cc11-459a-9efa-93df536aba61\") "
Jul 17 05:44:06.744206 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:06.744184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711a5ee2-cc11-459a-9efa-93df536aba61-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "711a5ee2-cc11-459a-9efa-93df536aba61" (UID: "711a5ee2-cc11-459a-9efa-93df536aba61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:44:06.845202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:06.845163    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711a5ee2-cc11-459a-9efa-93df536aba61-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:44:07.448716 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:07.448682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc" event={"ID":"711a5ee2-cc11-459a-9efa-93df536aba61","Type":"ContainerDied","Data":"e7cb0216b6d41b92c40720041ed17dd1e0f33ab3278fb8650de406c81aaf1f2d"}
Jul 17 05:44:07.448926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:07.448728    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"
Jul 17 05:44:07.448926 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:07.448750    2582 scope.go:117] "RemoveContainer" containerID="0f094298b0b330ead799acaefd029601963b40376eb908c31f120aa299770bed"
Jul 17 05:44:07.469052 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:07.469014    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"]
Jul 17 05:44:07.474159 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:07.474135    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-k55lc"]
Jul 17 05:44:08.730433 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:08.730404    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" path="/var/lib/kubelet/pods/711a5ee2-cc11-459a-9efa-93df536aba61/volumes"
Jul 17 05:44:11.446788 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.446754    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"]
Jul 17 05:44:11.447171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.447056    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:11.447171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.447068    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:11.447171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.447089    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:11.447171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.447094    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:11.447171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.447142    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:11.447171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.447150    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:11.451364 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.451347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:11.457382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.457360    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"]
Jul 17 05:44:11.589993 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.589950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/155772be-6dfd-420a-b069-a8f654d4c836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-h9d4g\" (UID: \"155772be-6dfd-420a-b069-a8f654d4c836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:11.691499 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.691454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/155772be-6dfd-420a-b069-a8f654d4c836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-h9d4g\" (UID: \"155772be-6dfd-420a-b069-a8f654d4c836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:11.692065 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.692046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/155772be-6dfd-420a-b069-a8f654d4c836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-h9d4g\" (UID: \"155772be-6dfd-420a-b069-a8f654d4c836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:11.724901 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.724829    2582 scope.go:117] "RemoveContainer" containerID="4c2d9fdb815b55ca9510e76c11d619334a9b8b41ed9ae9e9c0d1e9aaf0a1b916"
Jul 17 05:44:11.762997 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.762973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:11.893851 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:11.893821    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"]
Jul 17 05:44:11.895991 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:44:11.895967    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155772be_6dfd_420a_b069_a8f654d4c836.slice/crio-90300186c488ccd79f38d21675ba3c12c4b8c099bcc1303038f3a7b32ea8767e WatchSource:0}: Error finding container 90300186c488ccd79f38d21675ba3c12c4b8c099bcc1303038f3a7b32ea8767e: Status 404 returned error can't find the container with id 90300186c488ccd79f38d21675ba3c12c4b8c099bcc1303038f3a7b32ea8767e
Jul 17 05:44:12.418609 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:12.418572    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20bbeb9_471c_4de7_88ca_a28f3297e7d4.slice/crio-conmon-4c738bb4f3cbb81d04ddf2a35e25725c65bcc01968e8edc50d9196f695107037.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 05:44:12.474360 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.474333    2582 generic.go:356] "Generic (PLEG): container finished" podID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerID="4c738bb4f3cbb81d04ddf2a35e25725c65bcc01968e8edc50d9196f695107037" exitCode=6
Jul 17 05:44:12.474767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.474408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" event={"ID":"f20bbeb9-471c-4de7-88ca-a28f3297e7d4","Type":"ContainerDied","Data":"4c738bb4f3cbb81d04ddf2a35e25725c65bcc01968e8edc50d9196f695107037"}
Jul 17 05:44:12.474767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.474469    2582 scope.go:117] "RemoveContainer" containerID="4c2d9fdb815b55ca9510e76c11d619334a9b8b41ed9ae9e9c0d1e9aaf0a1b916"
Jul 17 05:44:12.474883 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.474828    2582 scope.go:117] "RemoveContainer" containerID="4c738bb4f3cbb81d04ddf2a35e25725c65bcc01968e8edc50d9196f695107037"
Jul 17 05:44:12.475080 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:12.475058    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh_odh-ai-gateway-infra(f20bbeb9-471c-4de7-88ca-a28f3297e7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4"
Jul 17 05:44:12.476005 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.475981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" event={"ID":"155772be-6dfd-420a-b069-a8f654d4c836","Type":"ContainerStarted","Data":"27d5a9e45b682c1bfa27b5ba1d61e2800ce84b48d29d7b11014423355aee0a14"}
Jul 17 05:44:12.476106 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.476031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" event={"ID":"155772be-6dfd-420a-b069-a8f654d4c836","Type":"ContainerStarted","Data":"90300186c488ccd79f38d21675ba3c12c4b8c099bcc1303038f3a7b32ea8767e"}
Jul 17 05:44:12.503722 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:12.503679    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" podStartSLOduration=1.503664314 podStartE2EDuration="1.503664314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:44:12.501883197 +0000 UTC m=+3256.238945533" watchObservedRunningTime="2026-07-17 05:44:12.503664314 +0000 UTC m=+3256.240726650"
Jul 17 05:44:13.483493 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.483452    2582 generic.go:356] "Generic (PLEG): container finished" podID="155772be-6dfd-420a-b069-a8f654d4c836" containerID="27d5a9e45b682c1bfa27b5ba1d61e2800ce84b48d29d7b11014423355aee0a14" exitCode=6
Jul 17 05:44:13.483867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.483511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" event={"ID":"155772be-6dfd-420a-b069-a8f654d4c836","Type":"ContainerDied","Data":"27d5a9e45b682c1bfa27b5ba1d61e2800ce84b48d29d7b11014423355aee0a14"}
Jul 17 05:44:13.483867 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.483729    2582 scope.go:117] "RemoveContainer" containerID="27d5a9e45b682c1bfa27b5ba1d61e2800ce84b48d29d7b11014423355aee0a14"
Jul 17 05:44:13.499557 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.499532    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"]
Jul 17 05:44:13.633580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.633557    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:44:13.809203 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.809173    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20bbeb9-471c-4de7-88ca-a28f3297e7d4-maas-api-service-ca\") pod \"f20bbeb9-471c-4de7-88ca-a28f3297e7d4\" (UID: \"f20bbeb9-471c-4de7-88ca-a28f3297e7d4\") "
Jul 17 05:44:13.809508 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.809487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20bbeb9-471c-4de7-88ca-a28f3297e7d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f20bbeb9-471c-4de7-88ca-a28f3297e7d4" (UID: "f20bbeb9-471c-4de7-88ca-a28f3297e7d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:44:13.910309 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:13.910279    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20bbeb9-471c-4de7-88ca-a28f3297e7d4-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:44:14.489592 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.489569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"
Jul 17 05:44:14.489592 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.489578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh" event={"ID":"f20bbeb9-471c-4de7-88ca-a28f3297e7d4","Type":"ContainerDied","Data":"fedbac69cfe22e49eb86d844e29ddc2d0bc2b5a82766267b50337e85454f1034"}
Jul 17 05:44:14.490108 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.489648    2582 scope.go:117] "RemoveContainer" containerID="4c738bb4f3cbb81d04ddf2a35e25725c65bcc01968e8edc50d9196f695107037"
Jul 17 05:44:14.491584 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.491563    2582 generic.go:356] "Generic (PLEG): container finished" podID="155772be-6dfd-420a-b069-a8f654d4c836" containerID="0bf60aff65607a7085bca786bbf847cfa89e37147b5fdeb85c8565c335074d2d" exitCode=6
Jul 17 05:44:14.491695 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.491654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" event={"ID":"155772be-6dfd-420a-b069-a8f654d4c836","Type":"ContainerDied","Data":"0bf60aff65607a7085bca786bbf847cfa89e37147b5fdeb85c8565c335074d2d"}
Jul 17 05:44:14.491908 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.491892    2582 scope.go:117] "RemoveContainer" containerID="0bf60aff65607a7085bca786bbf847cfa89e37147b5fdeb85c8565c335074d2d"
Jul 17 05:44:14.492173 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:14.492152    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-h9d4g_odh-ai-gateway-infra(155772be-6dfd-420a-b069-a8f654d4c836)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" podUID="155772be-6dfd-420a-b069-a8f654d4c836"
Jul 17 05:44:14.498807 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.498793    2582 scope.go:117] "RemoveContainer" containerID="27d5a9e45b682c1bfa27b5ba1d61e2800ce84b48d29d7b11014423355aee0a14"
Jul 17 05:44:14.521536 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.521507    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"]
Jul 17 05:44:14.523861 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.523832    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-hbbsh"]
Jul 17 05:44:14.729764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:14.729731    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" path="/var/lib/kubelet/pods/f20bbeb9-471c-4de7-88ca-a28f3297e7d4/volumes"
Jul 17 05:44:15.498669 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.498641    2582 scope.go:117] "RemoveContainer" containerID="0bf60aff65607a7085bca786bbf847cfa89e37147b5fdeb85c8565c335074d2d"
Jul 17 05:44:15.499074 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:15.498820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-h9d4g_odh-ai-gateway-infra(155772be-6dfd-420a-b069-a8f654d4c836)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" podUID="155772be-6dfd-420a-b069-a8f654d4c836"
Jul 17 05:44:15.534397 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534363    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"]
Jul 17 05:44:15.534669 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534658    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.534711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534671    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.534711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534682    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:15.534711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534688    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:15.534711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534707    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.534837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534713    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.534837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534758    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="711a5ee2-cc11-459a-9efa-93df536aba61" containerName="revoke-keys"
Jul 17 05:44:15.534837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534766    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.534837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534772    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.534837 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.534778    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:15.547168 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.547127    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"]
Jul 17 05:44:15.547363 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.547216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.549881 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.549857    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-823cda\"/\"default-dockercfg-9xxzq\""
Jul 17 05:44:15.550035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.549907    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-823cda\"/\"openshift-service-ca.crt\""
Jul 17 05:44:15.550035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.549864    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-823cda\"/\"kube-root-ca.crt\""
Jul 17 05:44:15.550035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.549877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-823cda\"/\"rate-test-model-823cda-kserve-self-signed-certs\""
Jul 17 05:44:15.624388 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.624348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-model-cache\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.624388 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.624392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a07a0a2-1258-47b5-8666-665661dd7e0d-tls-certs\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.624607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.624445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-dshm\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.624607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.624466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pplk\" (UniqueName: \"kubernetes.io/projected/7a07a0a2-1258-47b5-8666-665661dd7e0d-kube-api-access-8pplk\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.624607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.624532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-home\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.624607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.624565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-tmp-dir\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.725522 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.725483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pplk\" (UniqueName: \"kubernetes.io/projected/7a07a0a2-1258-47b5-8666-665661dd7e0d-kube-api-access-8pplk\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.725679 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.725532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-home\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.725679 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.725568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-tmp-dir\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.725679 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.725597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-model-cache\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.725679 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.725628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a07a0a2-1258-47b5-8666-665661dd7e0d-tls-certs\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.725679 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.725677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-dshm\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.726032 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.726008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-home\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.726092 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.726031    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-model-cache\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.726092 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.726075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-tmp-dir\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.728186 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.728165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-dshm\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.728382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.728365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a07a0a2-1258-47b5-8666-665661dd7e0d-tls-certs\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.733141 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.733118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pplk\" (UniqueName: \"kubernetes.io/projected/7a07a0a2-1258-47b5-8666-665661dd7e0d-kube-api-access-8pplk\") pod \"rate-test-model-823cda-kserve-5f96bc8cc9-bf25w\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") " pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.856283 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.856172    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:15.990382 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:15.990355    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"]
Jul 17 05:44:15.992767 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:44:15.992744    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a07a0a2_1258_47b5_8666_665661dd7e0d.slice/crio-6c9bfdbe45f3119d1ea407a6b207bfa8db34c525cf202c202ebadeb6cfc9a061 WatchSource:0}: Error finding container 6c9bfdbe45f3119d1ea407a6b207bfa8db34c525cf202c202ebadeb6cfc9a061: Status 404 returned error can't find the container with id 6c9bfdbe45f3119d1ea407a6b207bfa8db34c525cf202c202ebadeb6cfc9a061
Jul 17 05:44:16.504778 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:16.504744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w" event={"ID":"7a07a0a2-1258-47b5-8666-665661dd7e0d","Type":"ContainerStarted","Data":"c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785"}
Jul 17 05:44:16.504778 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:16.504794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w" event={"ID":"7a07a0a2-1258-47b5-8666-665661dd7e0d","Type":"ContainerStarted","Data":"6c9bfdbe45f3119d1ea407a6b207bfa8db34c525cf202c202ebadeb6cfc9a061"}
Jul 17 05:44:16.521859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:16.521796    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w" podStartSLOduration=1.5217808179999999 podStartE2EDuration="1.521780818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:44:16.519921009 +0000 UTC m=+3260.256983344" watchObservedRunningTime="2026-07-17 05:44:16.521780818 +0000 UTC m=+3260.258843155"
Jul 17 05:44:25.856814 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:25.856767    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:25.857356 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:25.856842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:25.869438 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:25.869413    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:26.563112 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:26.563081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:44:29.725048 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:29.725008    2582 scope.go:117] "RemoveContainer" containerID="0bf60aff65607a7085bca786bbf847cfa89e37147b5fdeb85c8565c335074d2d"
Jul 17 05:44:30.573499 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:30.573461    2582 generic.go:356] "Generic (PLEG): container finished" podID="155772be-6dfd-420a-b069-a8f654d4c836" containerID="94cc910a6f507806efa047537a9ec6ef725a3d60f52a2f2549821c920ae46fea" exitCode=6
Jul 17 05:44:30.573673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:30.573532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" event={"ID":"155772be-6dfd-420a-b069-a8f654d4c836","Type":"ContainerDied","Data":"94cc910a6f507806efa047537a9ec6ef725a3d60f52a2f2549821c920ae46fea"}
Jul 17 05:44:30.573673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:30.573593    2582 scope.go:117] "RemoveContainer" containerID="0bf60aff65607a7085bca786bbf847cfa89e37147b5fdeb85c8565c335074d2d"
Jul 17 05:44:30.573898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:30.573871    2582 scope.go:117] "RemoveContainer" containerID="94cc910a6f507806efa047537a9ec6ef725a3d60f52a2f2549821c920ae46fea"
Jul 17 05:44:30.574104 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:30.574086    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-h9d4g_odh-ai-gateway-infra(155772be-6dfd-420a-b069-a8f654d4c836)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" podUID="155772be-6dfd-420a-b069-a8f654d4c836"
Jul 17 05:44:31.598783 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:31.598742    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"]
Jul 17 05:44:31.727999 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:31.727974    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:31.729840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:31.729818    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/155772be-6dfd-420a-b069-a8f654d4c836-maas-api-service-ca\") pod \"155772be-6dfd-420a-b069-a8f654d4c836\" (UID: \"155772be-6dfd-420a-b069-a8f654d4c836\") "
Jul 17 05:44:31.730150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:31.730131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155772be-6dfd-420a-b069-a8f654d4c836-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "155772be-6dfd-420a-b069-a8f654d4c836" (UID: "155772be-6dfd-420a-b069-a8f654d4c836"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:44:31.830545 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:31.830504    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/155772be-6dfd-420a-b069-a8f654d4c836-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:44:32.585568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:32.585532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g" event={"ID":"155772be-6dfd-420a-b069-a8f654d4c836","Type":"ContainerDied","Data":"90300186c488ccd79f38d21675ba3c12c4b8c099bcc1303038f3a7b32ea8767e"}
Jul 17 05:44:32.585780 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:32.585587    2582 scope.go:117] "RemoveContainer" containerID="94cc910a6f507806efa047537a9ec6ef725a3d60f52a2f2549821c920ae46fea"
Jul 17 05:44:32.585780 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:32.585540    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"
Jul 17 05:44:32.605589 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:32.605566    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"]
Jul 17 05:44:32.609109 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:32.609086    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-h9d4g"]
Jul 17 05:44:32.729856 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:32.729822    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="155772be-6dfd-420a-b069-a8f654d4c836" path="/var/lib/kubelet/pods/155772be-6dfd-420a-b069-a8f654d4c836/volumes"
Jul 17 05:44:39.608036 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.607998    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"]
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608308    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608319    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20bbeb9-471c-4de7-88ca-a28f3297e7d4" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608327    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608333    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608349    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608354    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608407    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:39.608456 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.608418    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:39.611195 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.611179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:39.613507 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.613481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:44:39.617905 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.617879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"]
Jul 17 05:44:39.690635 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.690581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c0a4f-fd02-437d-84de-9367f3f2d743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp\" (UID: \"a12c0a4f-fd02-437d-84de-9367f3f2d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:39.791424 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.791383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c0a4f-fd02-437d-84de-9367f3f2d743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp\" (UID: \"a12c0a4f-fd02-437d-84de-9367f3f2d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:39.792015 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.791996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c0a4f-fd02-437d-84de-9367f3f2d743-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp\" (UID: \"a12c0a4f-fd02-437d-84de-9367f3f2d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:39.922711 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:39.922633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:40.055923 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:40.055897    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"]
Jul 17 05:44:40.058275 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:44:40.058237    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12c0a4f_fd02_437d_84de_9367f3f2d743.slice/crio-1023ccf3f3408c771a0ee6e93037b786bb0fa2356b8028df62517dc877376ed9 WatchSource:0}: Error finding container 1023ccf3f3408c771a0ee6e93037b786bb0fa2356b8028df62517dc877376ed9: Status 404 returned error can't find the container with id 1023ccf3f3408c771a0ee6e93037b786bb0fa2356b8028df62517dc877376ed9
Jul 17 05:44:40.627571 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:40.627483    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" event={"ID":"a12c0a4f-fd02-437d-84de-9367f3f2d743","Type":"ContainerStarted","Data":"79851031474b9a664a7b00ea2049cee7e9eed6c823cf434b16aafd5b87eb2e3b"}
Jul 17 05:44:40.627571 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:40.627538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" event={"ID":"a12c0a4f-fd02-437d-84de-9367f3f2d743","Type":"ContainerStarted","Data":"1023ccf3f3408c771a0ee6e93037b786bb0fa2356b8028df62517dc877376ed9"}
Jul 17 05:44:40.643512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:40.643456    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" podStartSLOduration=1.6434411500000001 podStartE2EDuration="1.64344115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:44:40.64218118 +0000 UTC m=+3284.379243515" watchObservedRunningTime="2026-07-17 05:44:40.64344115 +0000 UTC m=+3284.380503502"
Jul 17 05:44:41.633756 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:41.633719    2582 generic.go:356] "Generic (PLEG): container finished" podID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerID="79851031474b9a664a7b00ea2049cee7e9eed6c823cf434b16aafd5b87eb2e3b" exitCode=6
Jul 17 05:44:41.634239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:41.633802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" event={"ID":"a12c0a4f-fd02-437d-84de-9367f3f2d743","Type":"ContainerDied","Data":"79851031474b9a664a7b00ea2049cee7e9eed6c823cf434b16aafd5b87eb2e3b"}
Jul 17 05:44:41.634239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:41.634029    2582 scope.go:117] "RemoveContainer" containerID="79851031474b9a664a7b00ea2049cee7e9eed6c823cf434b16aafd5b87eb2e3b"
Jul 17 05:44:42.640580 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:42.640549    2582 generic.go:356] "Generic (PLEG): container finished" podID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerID="bf201d185efba001796851137cd02677f96fabe4689b831c4b56d83f22fb4c28" exitCode=6
Jul 17 05:44:42.641080 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:42.640627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" event={"ID":"a12c0a4f-fd02-437d-84de-9367f3f2d743","Type":"ContainerDied","Data":"bf201d185efba001796851137cd02677f96fabe4689b831c4b56d83f22fb4c28"}
Jul 17 05:44:42.641080 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:42.640677    2582 scope.go:117] "RemoveContainer" containerID="79851031474b9a664a7b00ea2049cee7e9eed6c823cf434b16aafd5b87eb2e3b"
Jul 17 05:44:42.641080 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:42.640892    2582 scope.go:117] "RemoveContainer" containerID="bf201d185efba001796851137cd02677f96fabe4689b831c4b56d83f22fb4c28"
Jul 17 05:44:42.641286 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:42.641125    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp_odh-ai-gateway-infra(a12c0a4f-fd02-437d-84de-9367f3f2d743)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743"
Jul 17 05:44:43.646969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:43.646943    2582 scope.go:117] "RemoveContainer" containerID="bf201d185efba001796851137cd02677f96fabe4689b831c4b56d83f22fb4c28"
Jul 17 05:44:43.647402 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:43.647125    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp_odh-ai-gateway-infra(a12c0a4f-fd02-437d-84de-9367f3f2d743)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743"
Jul 17 05:44:45.024150 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.024110    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"]
Jul 17 05:44:45.024566 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.024498    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:45.024566 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.024515    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:45.024641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.024584    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="155772be-6dfd-420a-b069-a8f654d4c836" containerName="revoke-keys"
Jul 17 05:44:45.027514 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.027497    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:44:45.033855 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.033827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"]
Jul 17 05:44:45.141209 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.141173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88277271-5f0c-45cd-a2b9-005bca7b51fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9\" (UID: \"88277271-5f0c-45cd-a2b9-005bca7b51fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:44:45.242552 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.242509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88277271-5f0c-45cd-a2b9-005bca7b51fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9\" (UID: \"88277271-5f0c-45cd-a2b9-005bca7b51fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:44:45.243148 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.243128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88277271-5f0c-45cd-a2b9-005bca7b51fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9\" (UID: \"88277271-5f0c-45cd-a2b9-005bca7b51fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:44:45.338606 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.338530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:44:45.492814 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.492780    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"]
Jul 17 05:44:45.495988 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:44:45.495949    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88277271_5f0c_45cd_a2b9_005bca7b51fd.slice/crio-162d1a2b451d01c4681ef8469462d8b26e9c32fd312c576e5359cb6002f7de93 WatchSource:0}: Error finding container 162d1a2b451d01c4681ef8469462d8b26e9c32fd312c576e5359cb6002f7de93: Status 404 returned error can't find the container with id 162d1a2b451d01c4681ef8469462d8b26e9c32fd312c576e5359cb6002f7de93
Jul 17 05:44:45.657819 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:45.657734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" event={"ID":"88277271-5f0c-45cd-a2b9-005bca7b51fd","Type":"ContainerStarted","Data":"162d1a2b451d01c4681ef8469462d8b26e9c32fd312c576e5359cb6002f7de93"}
Jul 17 05:44:46.664029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:46.663992    2582 generic.go:356] "Generic (PLEG): container finished" podID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerID="96b53518241cafc48ab37f47f5cdc7664ac43a44f41c71a2b34877943a1de2c0" exitCode=6
Jul 17 05:44:46.664535 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:46.664046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" event={"ID":"88277271-5f0c-45cd-a2b9-005bca7b51fd","Type":"ContainerDied","Data":"96b53518241cafc48ab37f47f5cdc7664ac43a44f41c71a2b34877943a1de2c0"}
Jul 17 05:44:46.664535 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:46.664297    2582 scope.go:117] "RemoveContainer" containerID="96b53518241cafc48ab37f47f5cdc7664ac43a44f41c71a2b34877943a1de2c0"
Jul 17 05:44:47.670523 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:47.670433    2582 generic.go:356] "Generic (PLEG): container finished" podID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerID="61772d5b55cc6c8e97c79f23ba8de6915a8156cb4961d29ef759e2a98f59221a" exitCode=6
Jul 17 05:44:47.670523 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:47.670489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" event={"ID":"88277271-5f0c-45cd-a2b9-005bca7b51fd","Type":"ContainerDied","Data":"61772d5b55cc6c8e97c79f23ba8de6915a8156cb4961d29ef759e2a98f59221a"}
Jul 17 05:44:47.670987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:47.670551    2582 scope.go:117] "RemoveContainer" containerID="96b53518241cafc48ab37f47f5cdc7664ac43a44f41c71a2b34877943a1de2c0"
Jul 17 05:44:47.670987 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:47.670750    2582 scope.go:117] "RemoveContainer" containerID="61772d5b55cc6c8e97c79f23ba8de6915a8156cb4961d29ef759e2a98f59221a"
Jul 17 05:44:47.670987 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:47.670969    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9_odh-ai-gateway-infra(88277271-5f0c-45cd-a2b9-005bca7b51fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd"
Jul 17 05:44:48.676846 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:48.676817    2582 scope.go:117] "RemoveContainer" containerID="61772d5b55cc6c8e97c79f23ba8de6915a8156cb4961d29ef759e2a98f59221a"
Jul 17 05:44:48.677224 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:48.677007    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9_odh-ai-gateway-infra(88277271-5f0c-45cd-a2b9-005bca7b51fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd"
Jul 17 05:44:54.724661 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:54.724634    2582 scope.go:117] "RemoveContainer" containerID="bf201d185efba001796851137cd02677f96fabe4689b831c4b56d83f22fb4c28"
Jul 17 05:44:55.712655 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:55.712623    2582 generic.go:356] "Generic (PLEG): container finished" podID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerID="e87de2ee148cf2b5807256091c055675c547c810ae373ddf7594690a90aee978" exitCode=6
Jul 17 05:44:55.712828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:55.712695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" event={"ID":"a12c0a4f-fd02-437d-84de-9367f3f2d743","Type":"ContainerDied","Data":"e87de2ee148cf2b5807256091c055675c547c810ae373ddf7594690a90aee978"}
Jul 17 05:44:55.712828 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:55.712772    2582 scope.go:117] "RemoveContainer" containerID="bf201d185efba001796851137cd02677f96fabe4689b831c4b56d83f22fb4c28"
Jul 17 05:44:55.713048 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:55.713027    2582 scope.go:117] "RemoveContainer" containerID="e87de2ee148cf2b5807256091c055675c547c810ae373ddf7594690a90aee978"
Jul 17 05:44:55.713279 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:44:55.713237    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp_odh-ai-gateway-infra(a12c0a4f-fd02-437d-84de-9367f3f2d743)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743"
Jul 17 05:44:56.736593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:56.736564    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"]
Jul 17 05:44:56.859460 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:56.859438    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:56.892558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:56.892530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:44:56.899079 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:56.899058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nxz9q_73058375-9228-4e88-93c4-3875b9c1ba27/ovn-acl-logging/0.log"
Jul 17 05:44:56.943640 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:56.943606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c0a4f-fd02-437d-84de-9367f3f2d743-maas-api-service-ca\") pod \"a12c0a4f-fd02-437d-84de-9367f3f2d743\" (UID: \"a12c0a4f-fd02-437d-84de-9367f3f2d743\") "
Jul 17 05:44:56.943982 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:56.943959    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12c0a4f-fd02-437d-84de-9367f3f2d743-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a12c0a4f-fd02-437d-84de-9367f3f2d743" (UID: "a12c0a4f-fd02-437d-84de-9367f3f2d743"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:44:57.044963 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:57.044888    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12c0a4f-fd02-437d-84de-9367f3f2d743-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:44:57.724973 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:57.724937    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"
Jul 17 05:44:57.724973 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:57.724947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp" event={"ID":"a12c0a4f-fd02-437d-84de-9367f3f2d743","Type":"ContainerDied","Data":"1023ccf3f3408c771a0ee6e93037b786bb0fa2356b8028df62517dc877376ed9"}
Jul 17 05:44:57.725281 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:57.725007    2582 scope.go:117] "RemoveContainer" containerID="e87de2ee148cf2b5807256091c055675c547c810ae373ddf7594690a90aee978"
Jul 17 05:44:57.745376 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:57.745347    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"]
Jul 17 05:44:57.754744 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:57.754719    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-n7khp"]
Jul 17 05:44:58.730099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:58.730067    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" path="/var/lib/kubelet/pods/a12c0a4f-fd02-437d-84de-9367f3f2d743/volumes"
Jul 17 05:44:59.309097 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:44:59.309060    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"]
Jul 17 05:45:01.724588 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:01.724550    2582 scope.go:117] "RemoveContainer" containerID="61772d5b55cc6c8e97c79f23ba8de6915a8156cb4961d29ef759e2a98f59221a"
Jul 17 05:45:02.752691 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:02.752660    2582 generic.go:356] "Generic (PLEG): container finished" podID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerID="72d8b2640eccf0298b83888c47df9cb55303b33a48fdfeb7750bcc245a540b97" exitCode=6
Jul 17 05:45:02.753156 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:02.752747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" event={"ID":"88277271-5f0c-45cd-a2b9-005bca7b51fd","Type":"ContainerDied","Data":"72d8b2640eccf0298b83888c47df9cb55303b33a48fdfeb7750bcc245a540b97"}
Jul 17 05:45:02.753156 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:02.752820    2582 scope.go:117] "RemoveContainer" containerID="61772d5b55cc6c8e97c79f23ba8de6915a8156cb4961d29ef759e2a98f59221a"
Jul 17 05:45:02.753156 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:02.753137    2582 scope.go:117] "RemoveContainer" containerID="72d8b2640eccf0298b83888c47df9cb55303b33a48fdfeb7750bcc245a540b97"
Jul 17 05:45:02.753362 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:02.753342    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9_odh-ai-gateway-infra(88277271-5f0c-45cd-a2b9-005bca7b51fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd"
Jul 17 05:45:03.176304 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:03.176198    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"]
Jul 17 05:45:03.179151 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:03.179124    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737740-wz2tf"]
Jul 17 05:45:03.778848 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:03.778807    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"]
Jul 17 05:45:03.915779 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:03.915754    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:45:04.008483 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.008447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88277271-5f0c-45cd-a2b9-005bca7b51fd-maas-api-service-ca\") pod \"88277271-5f0c-45cd-a2b9-005bca7b51fd\" (UID: \"88277271-5f0c-45cd-a2b9-005bca7b51fd\") "
Jul 17 05:45:04.008816 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.008793    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88277271-5f0c-45cd-a2b9-005bca7b51fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88277271-5f0c-45cd-a2b9-005bca7b51fd" (UID: "88277271-5f0c-45cd-a2b9-005bca7b51fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:04.109173 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.109094    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88277271-5f0c-45cd-a2b9-005bca7b51fd-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:04.730087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.730055    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8848042c-f164-4766-8ed3-006ae73fa1bc" path="/var/lib/kubelet/pods/8848042c-f164-4766-8ed3-006ae73fa1bc/volumes"
Jul 17 05:45:04.763613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.763583    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"
Jul 17 05:45:04.763613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.763603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9" event={"ID":"88277271-5f0c-45cd-a2b9-005bca7b51fd","Type":"ContainerDied","Data":"162d1a2b451d01c4681ef8469462d8b26e9c32fd312c576e5359cb6002f7de93"}
Jul 17 05:45:04.763826 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.763654    2582 scope.go:117] "RemoveContainer" containerID="72d8b2640eccf0298b83888c47df9cb55303b33a48fdfeb7750bcc245a540b97"
Jul 17 05:45:04.778171 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.778142    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"]
Jul 17 05:45:04.781754 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:04.781730    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-t86v9"]
Jul 17 05:45:06.729607 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:06.729573    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" path="/var/lib/kubelet/pods/88277271-5f0c-45cd-a2b9-005bca7b51fd/volumes"
Jul 17 05:45:14.329641 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.329583    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w" podUID="7a07a0a2-1258-47b5-8666-665661dd7e0d" containerName="main" containerID="cri-o://c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785" gracePeriod=45
Jul 17 05:45:14.574789 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.574766    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:45:14.698588 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-model-cache\") pod \"7a07a0a2-1258-47b5-8666-665661dd7e0d\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") "
Jul 17 05:45:14.698781 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-dshm\") pod \"7a07a0a2-1258-47b5-8666-665661dd7e0d\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") "
Jul 17 05:45:14.698781 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pplk\" (UniqueName: \"kubernetes.io/projected/7a07a0a2-1258-47b5-8666-665661dd7e0d-kube-api-access-8pplk\") pod \"7a07a0a2-1258-47b5-8666-665661dd7e0d\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") "
Jul 17 05:45:14.698781 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-home\") pod \"7a07a0a2-1258-47b5-8666-665661dd7e0d\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") "
Jul 17 05:45:14.698781 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-tmp-dir\") pod \"7a07a0a2-1258-47b5-8666-665661dd7e0d\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") "
Jul 17 05:45:14.699006 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a07a0a2-1258-47b5-8666-665661dd7e0d-tls-certs\") pod \"7a07a0a2-1258-47b5-8666-665661dd7e0d\" (UID: \"7a07a0a2-1258-47b5-8666-665661dd7e0d\") "
Jul 17 05:45:14.699006 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.698882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a07a0a2-1258-47b5-8666-665661dd7e0d" (UID: "7a07a0a2-1258-47b5-8666-665661dd7e0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:45:14.699099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.699056    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-home" (OuterVolumeSpecName: "home") pod "7a07a0a2-1258-47b5-8666-665661dd7e0d" (UID: "7a07a0a2-1258-47b5-8666-665661dd7e0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:45:14.699099 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.699070    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a07a0a2-1258-47b5-8666-665661dd7e0d" (UID: "7a07a0a2-1258-47b5-8666-665661dd7e0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:45:14.699184 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.699102    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-model-cache\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:14.701040 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.701019    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-dshm" (OuterVolumeSpecName: "dshm") pod "7a07a0a2-1258-47b5-8666-665661dd7e0d" (UID: "7a07a0a2-1258-47b5-8666-665661dd7e0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 05:45:14.701126 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.701016    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a07a0a2-1258-47b5-8666-665661dd7e0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a07a0a2-1258-47b5-8666-665661dd7e0d" (UID: "7a07a0a2-1258-47b5-8666-665661dd7e0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:45:14.701165 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.701124    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a07a0a2-1258-47b5-8666-665661dd7e0d-kube-api-access-8pplk" (OuterVolumeSpecName: "kube-api-access-8pplk") pod "7a07a0a2-1258-47b5-8666-665661dd7e0d" (UID: "7a07a0a2-1258-47b5-8666-665661dd7e0d"). InnerVolumeSpecName "kube-api-access-8pplk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:45:14.800502 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.800465    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pplk\" (UniqueName: \"kubernetes.io/projected/7a07a0a2-1258-47b5-8666-665661dd7e0d-kube-api-access-8pplk\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:14.800502 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.800496    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-home\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:14.800502 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.800507    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-tmp-dir\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:14.800730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.800516    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a07a0a2-1258-47b5-8666-665661dd7e0d-tls-certs\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:14.800730 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.800523    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a07a0a2-1258-47b5-8666-665661dd7e0d-dshm\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:14.810249 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.810223    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a07a0a2-1258-47b5-8666-665661dd7e0d" containerID="c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785" exitCode=0
Jul 17 05:45:14.810422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.810291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w" event={"ID":"7a07a0a2-1258-47b5-8666-665661dd7e0d","Type":"ContainerDied","Data":"c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785"}
Jul 17 05:45:14.810422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.810347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w" event={"ID":"7a07a0a2-1258-47b5-8666-665661dd7e0d","Type":"ContainerDied","Data":"6c9bfdbe45f3119d1ea407a6b207bfa8db34c525cf202c202ebadeb6cfc9a061"}
Jul 17 05:45:14.810422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.810364    2582 scope.go:117] "RemoveContainer" containerID="c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785"
Jul 17 05:45:14.810422 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.810302    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"
Jul 17 05:45:14.819027 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.819010    2582 scope.go:117] "RemoveContainer" containerID="c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785"
Jul 17 05:45:14.819316 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:14.819296    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785\": container with ID starting with c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785 not found: ID does not exist" containerID="c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785"
Jul 17 05:45:14.819358 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.819323    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785"} err="failed to get container status \"c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785\": rpc error: code = NotFound desc = could not find container \"c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785\": container with ID starting with c0bc4c4bf4c44df7ec31087fd34b539978268866da0cf19a3b47658f69e37785 not found: ID does not exist"
Jul 17 05:45:14.827078 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.827055    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"]
Jul 17 05:45:14.830762 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:14.830742    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-823cda/rate-test-model-823cda-kserve-5f96bc8cc9-bf25w"]
Jul 17 05:45:16.730663 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:16.730631    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a07a0a2-1258-47b5-8666-665661dd7e0d" path="/var/lib/kubelet/pods/7a07a0a2-1258-47b5-8666-665661dd7e0d/volumes"
Jul 17 05:45:17.738784 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.738751    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"]
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a07a0a2-1258-47b5-8666-665661dd7e0d" containerName="main"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a07a0a2-1258-47b5-8666-665661dd7e0d" containerName="main"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739057    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739063    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739076    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739081    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739087    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739093    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739101    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739106    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739114    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739119    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739163    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:17.739569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739170    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:17.739569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739177    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739182    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a07a0a2-1258-47b5-8666-665661dd7e0d" containerName="main"
Jul 17 05:45:17.739569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739187    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.739569 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.739192    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12c0a4f-fd02-437d-84de-9367f3f2d743" containerName="revoke-keys"
Jul 17 05:45:17.748902 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.748873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:17.749625 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.749599    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"]
Jul 17 05:45:17.751658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.751638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:45:17.927985 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:17.927949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502fd029-c7d6-4bd8-af1a-d52a4592e21f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5\" (UID: \"502fd029-c7d6-4bd8-af1a-d52a4592e21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:18.029083 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.028976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502fd029-c7d6-4bd8-af1a-d52a4592e21f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5\" (UID: \"502fd029-c7d6-4bd8-af1a-d52a4592e21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:18.029707 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.029680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502fd029-c7d6-4bd8-af1a-d52a4592e21f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5\" (UID: \"502fd029-c7d6-4bd8-af1a-d52a4592e21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:18.060065 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.060032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:18.195135 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.195112    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"]
Jul 17 05:45:18.196998 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:45:18.196970    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502fd029_c7d6_4bd8_af1a_d52a4592e21f.slice/crio-a575ae78ba647d0cdc7d3f42716dbd3122ce6999b294ed3f4391f1549f4355ac WatchSource:0}: Error finding container a575ae78ba647d0cdc7d3f42716dbd3122ce6999b294ed3f4391f1549f4355ac: Status 404 returned error can't find the container with id a575ae78ba647d0cdc7d3f42716dbd3122ce6999b294ed3f4391f1549f4355ac
Jul 17 05:45:18.830763 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.830735    2582 generic.go:356] "Generic (PLEG): container finished" podID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerID="fa931feaadb10645d002cc4eb4760404f5a3fbb5b4a2843e1db06b035e822635" exitCode=6
Jul 17 05:45:18.831118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.830814    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" event={"ID":"502fd029-c7d6-4bd8-af1a-d52a4592e21f","Type":"ContainerDied","Data":"fa931feaadb10645d002cc4eb4760404f5a3fbb5b4a2843e1db06b035e822635"}
Jul 17 05:45:18.831118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.830867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" event={"ID":"502fd029-c7d6-4bd8-af1a-d52a4592e21f","Type":"ContainerStarted","Data":"a575ae78ba647d0cdc7d3f42716dbd3122ce6999b294ed3f4391f1549f4355ac"}
Jul 17 05:45:18.831118 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:18.831031    2582 scope.go:117] "RemoveContainer" containerID="fa931feaadb10645d002cc4eb4760404f5a3fbb5b4a2843e1db06b035e822635"
Jul 17 05:45:19.837839 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:19.837752    2582 generic.go:356] "Generic (PLEG): container finished" podID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerID="6c4eb4a0822f9aec4cdd29c62ac8e5cb21642eaa1b1a1bb7c1602f2e89ce21cf" exitCode=6
Jul 17 05:45:19.838241 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:19.837847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" event={"ID":"502fd029-c7d6-4bd8-af1a-d52a4592e21f","Type":"ContainerDied","Data":"6c4eb4a0822f9aec4cdd29c62ac8e5cb21642eaa1b1a1bb7c1602f2e89ce21cf"}
Jul 17 05:45:19.838241 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:19.837915    2582 scope.go:117] "RemoveContainer" containerID="fa931feaadb10645d002cc4eb4760404f5a3fbb5b4a2843e1db06b035e822635"
Jul 17 05:45:19.838241 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:19.838174    2582 scope.go:117] "RemoveContainer" containerID="6c4eb4a0822f9aec4cdd29c62ac8e5cb21642eaa1b1a1bb7c1602f2e89ce21cf"
Jul 17 05:45:19.838413 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:19.838398    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5_odh-ai-gateway-infra(502fd029-c7d6-4bd8-af1a-d52a4592e21f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f"
Jul 17 05:45:20.844772 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:20.844743    2582 scope.go:117] "RemoveContainer" containerID="6c4eb4a0822f9aec4cdd29c62ac8e5cb21642eaa1b1a1bb7c1602f2e89ce21cf"
Jul 17 05:45:20.845156 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:20.844951    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5_odh-ai-gateway-infra(502fd029-c7d6-4bd8-af1a-d52a4592e21f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f"
Jul 17 05:45:29.226728 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.226693    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"]
Jul 17 05:45:29.227248 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.226931    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh" podUID="7f138551-bf8a-464a-acd5-ade1e2c84f9d" containerName="maas-api" containerID="cri-o://c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915" gracePeriod=30
Jul 17 05:45:29.461545 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.461522    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:45:29.526065 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.525981    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f138551-bf8a-464a-acd5-ade1e2c84f9d-openshift-service-ca\") pod \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") "
Jul 17 05:45:29.526065 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.526045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f138551-bf8a-464a-acd5-ade1e2c84f9d-maas-api-tls\") pod \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") "
Jul 17 05:45:29.526300 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.526078    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdd2l\" (UniqueName: \"kubernetes.io/projected/7f138551-bf8a-464a-acd5-ade1e2c84f9d-kube-api-access-rdd2l\") pod \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\" (UID: \"7f138551-bf8a-464a-acd5-ade1e2c84f9d\") "
Jul 17 05:45:29.526388 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.526365    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f138551-bf8a-464a-acd5-ade1e2c84f9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f138551-bf8a-464a-acd5-ade1e2c84f9d" (UID: "7f138551-bf8a-464a-acd5-ade1e2c84f9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:29.528325 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.528299    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f138551-bf8a-464a-acd5-ade1e2c84f9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f138551-bf8a-464a-acd5-ade1e2c84f9d" (UID: "7f138551-bf8a-464a-acd5-ade1e2c84f9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:45:29.528428 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.528408    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f138551-bf8a-464a-acd5-ade1e2c84f9d-kube-api-access-rdd2l" (OuterVolumeSpecName: "kube-api-access-rdd2l") pod "7f138551-bf8a-464a-acd5-ade1e2c84f9d" (UID: "7f138551-bf8a-464a-acd5-ade1e2c84f9d"). InnerVolumeSpecName "kube-api-access-rdd2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:45:29.627327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.627242    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f138551-bf8a-464a-acd5-ade1e2c84f9d-maas-api-tls\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:29.627327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.627321    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdd2l\" (UniqueName: \"kubernetes.io/projected/7f138551-bf8a-464a-acd5-ade1e2c84f9d-kube-api-access-rdd2l\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:29.627327 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.627332    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f138551-bf8a-464a-acd5-ade1e2c84f9d-openshift-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:29.648577 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.648543    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"]
Jul 17 05:45:29.648946 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.648927    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:29.649037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.648950    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:29.649037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.648965    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f138551-bf8a-464a-acd5-ade1e2c84f9d" containerName="maas-api"
Jul 17 05:45:29.649037 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.648972    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f138551-bf8a-464a-acd5-ade1e2c84f9d" containerName="maas-api"
Jul 17 05:45:29.649191 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.649048    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88277271-5f0c-45cd-a2b9-005bca7b51fd" containerName="revoke-keys"
Jul 17 05:45:29.649191 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.649061    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f138551-bf8a-464a-acd5-ade1e2c84f9d" containerName="maas-api"
Jul 17 05:45:29.652103 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.652081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:29.659340 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.659253    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"]
Jul 17 05:45:29.728558 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.728515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22177f62-ffc1-40cc-8309-0d700c1cc9fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b\" (UID: \"22177f62-ffc1-40cc-8309-0d700c1cc9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:29.829873 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.829784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22177f62-ffc1-40cc-8309-0d700c1cc9fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b\" (UID: \"22177f62-ffc1-40cc-8309-0d700c1cc9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:29.830507 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.830490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22177f62-ffc1-40cc-8309-0d700c1cc9fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b\" (UID: \"22177f62-ffc1-40cc-8309-0d700c1cc9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:29.885950 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.885915    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f138551-bf8a-464a-acd5-ade1e2c84f9d" containerID="c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915" exitCode=0
Jul 17 05:45:29.886130 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.885965    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"
Jul 17 05:45:29.886130 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.886003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh" event={"ID":"7f138551-bf8a-464a-acd5-ade1e2c84f9d","Type":"ContainerDied","Data":"c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915"}
Jul 17 05:45:29.886130 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.886067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh" event={"ID":"7f138551-bf8a-464a-acd5-ade1e2c84f9d","Type":"ContainerDied","Data":"c3360203c2a179b16d3f11380df2331e983899cd81f504f226f638ee34afda66"}
Jul 17 05:45:29.886130 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.886097    2582 scope.go:117] "RemoveContainer" containerID="c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915"
Jul 17 05:45:29.901453 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.901432    2582 scope.go:117] "RemoveContainer" containerID="c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915"
Jul 17 05:45:29.901722 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:29.901703    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915\": container with ID starting with c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915 not found: ID does not exist" containerID="c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915"
Jul 17 05:45:29.901787 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.901735    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915"} err="failed to get container status \"c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915\": rpc error: code = NotFound desc = could not find container \"c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915\": container with ID starting with c0f30cf146ea213b2cad2a8477cabdc4d4ddfd6a610187432710aac9f61a1915 not found: ID does not exist"
Jul 17 05:45:29.913743 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.913706    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"]
Jul 17 05:45:29.917250 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.917226    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-560f5b-5899566df9-2hnwh"]
Jul 17 05:45:29.964453 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:29.964419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:30.091765 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:30.091735    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"]
Jul 17 05:45:30.093947 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:45:30.093923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22177f62_ffc1_40cc_8309_0d700c1cc9fa.slice/crio-6b6b9fca1a950872ebe5d9e2347dbedd554fbc57a1518a7cb1be460262b1a80c WatchSource:0}: Error finding container 6b6b9fca1a950872ebe5d9e2347dbedd554fbc57a1518a7cb1be460262b1a80c: Status 404 returned error can't find the container with id 6b6b9fca1a950872ebe5d9e2347dbedd554fbc57a1518a7cb1be460262b1a80c
Jul 17 05:45:30.729794 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:30.729763    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f138551-bf8a-464a-acd5-ade1e2c84f9d" path="/var/lib/kubelet/pods/7f138551-bf8a-464a-acd5-ade1e2c84f9d/volumes"
Jul 17 05:45:30.891898 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:30.891865    2582 generic.go:356] "Generic (PLEG): container finished" podID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerID="b3b299cb8c98dc3e190757f065fbf0dc4c3cd4ed43410504753a8566f780611e" exitCode=6
Jul 17 05:45:30.892091 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:30.891948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" event={"ID":"22177f62-ffc1-40cc-8309-0d700c1cc9fa","Type":"ContainerDied","Data":"b3b299cb8c98dc3e190757f065fbf0dc4c3cd4ed43410504753a8566f780611e"}
Jul 17 05:45:30.892091 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:30.891997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" event={"ID":"22177f62-ffc1-40cc-8309-0d700c1cc9fa","Type":"ContainerStarted","Data":"6b6b9fca1a950872ebe5d9e2347dbedd554fbc57a1518a7cb1be460262b1a80c"}
Jul 17 05:45:30.892207 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:30.892114    2582 scope.go:117] "RemoveContainer" containerID="b3b299cb8c98dc3e190757f065fbf0dc4c3cd4ed43410504753a8566f780611e"
Jul 17 05:45:31.898800 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:31.898716    2582 generic.go:356] "Generic (PLEG): container finished" podID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerID="3db084a450fe650fc8fe16f11ff6828972aaf10ccfe81dfef332575f981f0b45" exitCode=6
Jul 17 05:45:31.899212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:31.898808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" event={"ID":"22177f62-ffc1-40cc-8309-0d700c1cc9fa","Type":"ContainerDied","Data":"3db084a450fe650fc8fe16f11ff6828972aaf10ccfe81dfef332575f981f0b45"}
Jul 17 05:45:31.899212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:31.898872    2582 scope.go:117] "RemoveContainer" containerID="b3b299cb8c98dc3e190757f065fbf0dc4c3cd4ed43410504753a8566f780611e"
Jul 17 05:45:31.899212 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:31.898973    2582 scope.go:117] "RemoveContainer" containerID="3db084a450fe650fc8fe16f11ff6828972aaf10ccfe81dfef332575f981f0b45"
Jul 17 05:45:31.899369 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:31.899221    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b_odh-ai-gateway-infra(22177f62-ffc1-40cc-8309-0d700c1cc9fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa"
Jul 17 05:45:32.904555 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:32.904528    2582 scope.go:117] "RemoveContainer" containerID="3db084a450fe650fc8fe16f11ff6828972aaf10ccfe81dfef332575f981f0b45"
Jul 17 05:45:32.904935 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:32.904697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b_odh-ai-gateway-infra(22177f62-ffc1-40cc-8309-0d700c1cc9fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa"
Jul 17 05:45:34.725687 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:34.725657    2582 scope.go:117] "RemoveContainer" containerID="6c4eb4a0822f9aec4cdd29c62ac8e5cb21642eaa1b1a1bb7c1602f2e89ce21cf"
Jul 17 05:45:35.056975 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.056939    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"]
Jul 17 05:45:35.060144 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.060126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:35.066459 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.066435    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"]
Jul 17 05:45:35.077800 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.077777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1541ebeb-10e3-4ca8-b05d-e6de59fa726e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf\" (UID: \"1541ebeb-10e3-4ca8-b05d-e6de59fa726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:35.178992 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.178959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1541ebeb-10e3-4ca8-b05d-e6de59fa726e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf\" (UID: \"1541ebeb-10e3-4ca8-b05d-e6de59fa726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:35.179608 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.179590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1541ebeb-10e3-4ca8-b05d-e6de59fa726e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf\" (UID: \"1541ebeb-10e3-4ca8-b05d-e6de59fa726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:35.371661 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.371630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:35.504614 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.504583    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"]
Jul 17 05:45:35.507758 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:45:35.507730    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1541ebeb_10e3_4ca8_b05d_e6de59fa726e.slice/crio-59360d526549edb639607d747fff1d8096731b913e528ec312d6a33341b5c84c WatchSource:0}: Error finding container 59360d526549edb639607d747fff1d8096731b913e528ec312d6a33341b5c84c: Status 404 returned error can't find the container with id 59360d526549edb639607d747fff1d8096731b913e528ec312d6a33341b5c84c
Jul 17 05:45:35.919791 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.919749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerStarted","Data":"9ec679a4545c966ece020d4436e8af850336bedc36cce90d3dec45ecbbf4799b"}
Jul 17 05:45:35.920174 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.919808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerStarted","Data":"59360d526549edb639607d747fff1d8096731b913e528ec312d6a33341b5c84c"}
Jul 17 05:45:35.921621 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.921596    2582 generic.go:356] "Generic (PLEG): container finished" podID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerID="fa13478470475e5923a4b87850846b865fb7d6391bd6af6b018ebbb6de8708f2" exitCode=6
Jul 17 05:45:35.921758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.921632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" event={"ID":"502fd029-c7d6-4bd8-af1a-d52a4592e21f","Type":"ContainerDied","Data":"fa13478470475e5923a4b87850846b865fb7d6391bd6af6b018ebbb6de8708f2"}
Jul 17 05:45:35.921758 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.921671    2582 scope.go:117] "RemoveContainer" containerID="6c4eb4a0822f9aec4cdd29c62ac8e5cb21642eaa1b1a1bb7c1602f2e89ce21cf"
Jul 17 05:45:35.921958 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.921938    2582 scope.go:117] "RemoveContainer" containerID="fa13478470475e5923a4b87850846b865fb7d6391bd6af6b018ebbb6de8708f2"
Jul 17 05:45:35.922151 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:35.922125    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5_odh-ai-gateway-infra(502fd029-c7d6-4bd8-af1a-d52a4592e21f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f"
Jul 17 05:45:35.935479 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:35.935436    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" podStartSLOduration=0.935423889 podStartE2EDuration="935.423889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:35.933006743 +0000 UTC m=+3339.670069079" watchObservedRunningTime="2026-07-17 05:45:35.935423889 +0000 UTC m=+3339.672486225"
Jul 17 05:45:36.927739 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:36.927705    2582 generic.go:356] "Generic (PLEG): container finished" podID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerID="9ec679a4545c966ece020d4436e8af850336bedc36cce90d3dec45ecbbf4799b" exitCode=6
Jul 17 05:45:36.928196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:36.927797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerDied","Data":"9ec679a4545c966ece020d4436e8af850336bedc36cce90d3dec45ecbbf4799b"}
Jul 17 05:45:36.928196 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:36.927927    2582 scope.go:117] "RemoveContainer" containerID="9ec679a4545c966ece020d4436e8af850336bedc36cce90d3dec45ecbbf4799b"
Jul 17 05:45:36.957031 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:36.957002    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"]
Jul 17 05:45:37.091407 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.091383    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:37.195347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.195324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502fd029-c7d6-4bd8-af1a-d52a4592e21f-maas-api-service-ca\") pod \"502fd029-c7d6-4bd8-af1a-d52a4592e21f\" (UID: \"502fd029-c7d6-4bd8-af1a-d52a4592e21f\") "
Jul 17 05:45:37.195652 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.195631    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502fd029-c7d6-4bd8-af1a-d52a4592e21f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "502fd029-c7d6-4bd8-af1a-d52a4592e21f" (UID: "502fd029-c7d6-4bd8-af1a-d52a4592e21f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:37.296098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.296066    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502fd029-c7d6-4bd8-af1a-d52a4592e21f-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:37.935169 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.935092    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"
Jul 17 05:45:37.935649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.935095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5" event={"ID":"502fd029-c7d6-4bd8-af1a-d52a4592e21f","Type":"ContainerDied","Data":"a575ae78ba647d0cdc7d3f42716dbd3122ce6999b294ed3f4391f1549f4355ac"}
Jul 17 05:45:37.935649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.935227    2582 scope.go:117] "RemoveContainer" containerID="fa13478470475e5923a4b87850846b865fb7d6391bd6af6b018ebbb6de8708f2"
Jul 17 05:45:37.937194 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.937169    2582 generic.go:356] "Generic (PLEG): container finished" podID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerID="b1c708d01815af9412001758327fec57d4c92f7ec4adb9f70e8e0e9d18c78698" exitCode=6
Jul 17 05:45:37.937331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.937220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerDied","Data":"b1c708d01815af9412001758327fec57d4c92f7ec4adb9f70e8e0e9d18c78698"}
Jul 17 05:45:37.937518 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.937502    2582 scope.go:117] "RemoveContainer" containerID="b1c708d01815af9412001758327fec57d4c92f7ec4adb9f70e8e0e9d18c78698"
Jul 17 05:45:37.937762 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:37.937741    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf_odh-ai-gateway-infra(1541ebeb-10e3-4ca8-b05d-e6de59fa726e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e"
Jul 17 05:45:37.944380 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.944364    2582 scope.go:117] "RemoveContainer" containerID="9ec679a4545c966ece020d4436e8af850336bedc36cce90d3dec45ecbbf4799b"
Jul 17 05:45:37.968270 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.968234    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"]
Jul 17 05:45:37.973649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:37.973625    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eb8486e-bqpv5"]
Jul 17 05:45:38.731015 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:38.730982    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" path="/var/lib/kubelet/pods/502fd029-c7d6-4bd8-af1a-d52a4592e21f/volumes"
Jul 17 05:45:38.944208 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:38.944186    2582 scope.go:117] "RemoveContainer" containerID="b1c708d01815af9412001758327fec57d4c92f7ec4adb9f70e8e0e9d18c78698"
Jul 17 05:45:38.944596 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:38.944446    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf_odh-ai-gateway-infra(1541ebeb-10e3-4ca8-b05d-e6de59fa726e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e"
Jul 17 05:45:44.902501 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902470    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"]
Jul 17 05:45:44.902858 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902801    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:45:44.902858 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902817    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:45:44.902858 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902834    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:45:44.902858 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902839    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:45:44.903012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902899    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:45:44.903012 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.902912    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:45:44.907021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.907003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:44.912609 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.912573    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"]
Jul 17 05:45:44.959098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:44.959063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b73d8f-f964-4b49-bb59-d5acc612e15e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-823cda-g8txn\" (UID: \"71b73d8f-f964-4b49-bb59-d5acc612e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:45.059749 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.059705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b73d8f-f964-4b49-bb59-d5acc612e15e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-823cda-g8txn\" (UID: \"71b73d8f-f964-4b49-bb59-d5acc612e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:45.060337 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.060320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b73d8f-f964-4b49-bb59-d5acc612e15e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-823cda-g8txn\" (UID: \"71b73d8f-f964-4b49-bb59-d5acc612e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:45.218738 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.218699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:45.350486 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.350462    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"]
Jul 17 05:45:45.352928 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:45:45.352903    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b73d8f_f964_4b49_bb59_d5acc612e15e.slice/crio-e398c52095d7b31fbc204bbe374b1ba8b8eb56a792e15d51ff5f891bb3f0c0ca WatchSource:0}: Error finding container e398c52095d7b31fbc204bbe374b1ba8b8eb56a792e15d51ff5f891bb3f0c0ca: Status 404 returned error can't find the container with id e398c52095d7b31fbc204bbe374b1ba8b8eb56a792e15d51ff5f891bb3f0c0ca
Jul 17 05:45:45.979467 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.979430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn" event={"ID":"71b73d8f-f964-4b49-bb59-d5acc612e15e","Type":"ContainerStarted","Data":"52cbd1cbf9da34ef2213059e659233b49a32ccc8c14ff2238ba84b1f1f0fb19e"}
Jul 17 05:45:45.979840 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.979489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn" event={"ID":"71b73d8f-f964-4b49-bb59-d5acc612e15e","Type":"ContainerStarted","Data":"e398c52095d7b31fbc204bbe374b1ba8b8eb56a792e15d51ff5f891bb3f0c0ca"}
Jul 17 05:45:45.993281 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:45.993214    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn" podStartSLOduration=1.993196732 podStartE2EDuration="1.993196732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:45.992059052 +0000 UTC m=+3349.729121392" watchObservedRunningTime="2026-07-17 05:45:45.993196732 +0000 UTC m=+3349.730259068"
Jul 17 05:45:46.986384 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:46.986352    2582 generic.go:356] "Generic (PLEG): container finished" podID="71b73d8f-f964-4b49-bb59-d5acc612e15e" containerID="52cbd1cbf9da34ef2213059e659233b49a32ccc8c14ff2238ba84b1f1f0fb19e" exitCode=0
Jul 17 05:45:46.986818 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:46.986400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn" event={"ID":"71b73d8f-f964-4b49-bb59-d5acc612e15e","Type":"ContainerDied","Data":"52cbd1cbf9da34ef2213059e659233b49a32ccc8c14ff2238ba84b1f1f0fb19e"}
Jul 17 05:45:47.725419 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:47.725389    2582 scope.go:117] "RemoveContainer" containerID="3db084a450fe650fc8fe16f11ff6828972aaf10ccfe81dfef332575f981f0b45"
Jul 17 05:45:47.993273 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:47.993155    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" event={"ID":"22177f62-ffc1-40cc-8309-0d700c1cc9fa","Type":"ContainerStarted","Data":"559ccffe0a982eb353e59f3368f954c4fc6c122595a17f8aeced52ebf05a45f3"}
Jul 17 05:45:48.012052 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.011997    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" podStartSLOduration=19.011975343 podStartE2EDuration="19.011975343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:48.009782159 +0000 UTC m=+3351.746844494" watchObservedRunningTime="2026-07-17 05:45:48.011975343 +0000 UTC m=+3351.749037680"
Jul 17 05:45:48.132141 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.132116    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:48.185185 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.185146    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b73d8f-f964-4b49-bb59-d5acc612e15e-maas-api-service-ca\") pod \"71b73d8f-f964-4b49-bb59-d5acc612e15e\" (UID: \"71b73d8f-f964-4b49-bb59-d5acc612e15e\") "
Jul 17 05:45:48.185528 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.185503    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b73d8f-f964-4b49-bb59-d5acc612e15e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71b73d8f-f964-4b49-bb59-d5acc612e15e" (UID: "71b73d8f-f964-4b49-bb59-d5acc612e15e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:48.286074 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.286043    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b73d8f-f964-4b49-bb59-d5acc612e15e-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:48.999512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.999436    2582 generic.go:356] "Generic (PLEG): container finished" podID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerID="559ccffe0a982eb353e59f3368f954c4fc6c122595a17f8aeced52ebf05a45f3" exitCode=6
Jul 17 05:45:48.999512 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.999504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" event={"ID":"22177f62-ffc1-40cc-8309-0d700c1cc9fa","Type":"ContainerDied","Data":"559ccffe0a982eb353e59f3368f954c4fc6c122595a17f8aeced52ebf05a45f3"}
Jul 17 05:45:49.000016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.999566    2582 scope.go:117] "RemoveContainer" containerID="3db084a450fe650fc8fe16f11ff6828972aaf10ccfe81dfef332575f981f0b45"
Jul 17 05:45:49.000016 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:48.999846    2582 scope.go:117] "RemoveContainer" containerID="559ccffe0a982eb353e59f3368f954c4fc6c122595a17f8aeced52ebf05a45f3"
Jul 17 05:45:49.000133 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:49.000072    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b_odh-ai-gateway-infra(22177f62-ffc1-40cc-8309-0d700c1cc9fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa"
Jul 17 05:45:49.001145 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:49.001118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn" event={"ID":"71b73d8f-f964-4b49-bb59-d5acc612e15e","Type":"ContainerDied","Data":"e398c52095d7b31fbc204bbe374b1ba8b8eb56a792e15d51ff5f891bb3f0c0ca"}
Jul 17 05:45:49.001205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:49.001148    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-823cda-g8txn"
Jul 17 05:45:49.001205 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:49.001169    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e398c52095d7b31fbc204bbe374b1ba8b8eb56a792e15d51ff5f891bb3f0c0ca"
Jul 17 05:45:49.021590 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:49.021564    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"]
Jul 17 05:45:50.157526 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:50.157499    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:50.202676 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:50.202641    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22177f62-ffc1-40cc-8309-0d700c1cc9fa-maas-api-service-ca\") pod \"22177f62-ffc1-40cc-8309-0d700c1cc9fa\" (UID: \"22177f62-ffc1-40cc-8309-0d700c1cc9fa\") "
Jul 17 05:45:50.202969 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:50.202946    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22177f62-ffc1-40cc-8309-0d700c1cc9fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22177f62-ffc1-40cc-8309-0d700c1cc9fa" (UID: "22177f62-ffc1-40cc-8309-0d700c1cc9fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:50.304035 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:50.303954    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22177f62-ffc1-40cc-8309-0d700c1cc9fa-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:51.013406 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:51.013326    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"
Jul 17 05:45:51.013406 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:51.013360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b" event={"ID":"22177f62-ffc1-40cc-8309-0d700c1cc9fa","Type":"ContainerDied","Data":"6b6b9fca1a950872ebe5d9e2347dbedd554fbc57a1518a7cb1be460262b1a80c"}
Jul 17 05:45:51.013613 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:51.013421    2582 scope.go:117] "RemoveContainer" containerID="559ccffe0a982eb353e59f3368f954c4fc6c122595a17f8aeced52ebf05a45f3"
Jul 17 05:45:51.029014 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:51.028980    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"]
Jul 17 05:45:51.034372 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:51.034338    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8a383ce-jsd5b"]
Jul 17 05:45:52.729450 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:52.729418    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" path="/var/lib/kubelet/pods/22177f62-ffc1-40cc-8309-0d700c1cc9fa/volumes"
Jul 17 05:45:53.724568 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:53.724536    2582 scope.go:117] "RemoveContainer" containerID="b1c708d01815af9412001758327fec57d4c92f7ec4adb9f70e8e0e9d18c78698"
Jul 17 05:45:54.029492 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:54.029461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerStarted","Data":"b00ba9932a69d06b37c62f8f73ffb6d7e4af91ca30b240d7e35fbcd4a12a304b"}
Jul 17 05:45:55.035952 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.035915    2582 generic.go:356] "Generic (PLEG): container finished" podID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerID="b00ba9932a69d06b37c62f8f73ffb6d7e4af91ca30b240d7e35fbcd4a12a304b" exitCode=6
Jul 17 05:45:55.036420 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.035981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerDied","Data":"b00ba9932a69d06b37c62f8f73ffb6d7e4af91ca30b240d7e35fbcd4a12a304b"}
Jul 17 05:45:55.036420 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.036039    2582 scope.go:117] "RemoveContainer" containerID="b1c708d01815af9412001758327fec57d4c92f7ec4adb9f70e8e0e9d18c78698"
Jul 17 05:45:55.036420 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.036326    2582 scope.go:117] "RemoveContainer" containerID="b00ba9932a69d06b37c62f8f73ffb6d7e4af91ca30b240d7e35fbcd4a12a304b"
Jul 17 05:45:55.036552 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:55.036534    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf_odh-ai-gateway-infra(1541ebeb-10e3-4ca8-b05d-e6de59fa726e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e"
Jul 17 05:45:55.059510 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.059476    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"]
Jul 17 05:45:55.315767 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.315657    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"]
Jul 17 05:45:55.315936 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.315870    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" podUID="45a50ce9-2dfc-46eb-aaef-ea0d195ae141" containerName="maas-api" containerID="cri-o://036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256" gracePeriod=30
Jul 17 05:45:55.556123 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.556099    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:45:55.649859 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.649776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls\") pod \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") "
Jul 17 05:45:55.650021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.649887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr22n\" (UniqueName: \"kubernetes.io/projected/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-kube-api-access-zr22n\") pod \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") "
Jul 17 05:45:55.650021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.649906    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-openshift-service-ca\") pod \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\" (UID: \"45a50ce9-2dfc-46eb-aaef-ea0d195ae141\") "
Jul 17 05:45:55.650348 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.650321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45a50ce9-2dfc-46eb-aaef-ea0d195ae141" (UID: "45a50ce9-2dfc-46eb-aaef-ea0d195ae141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:55.652046 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.652015    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-kube-api-access-zr22n" (OuterVolumeSpecName: "kube-api-access-zr22n") pod "45a50ce9-2dfc-46eb-aaef-ea0d195ae141" (UID: "45a50ce9-2dfc-46eb-aaef-ea0d195ae141"). InnerVolumeSpecName "kube-api-access-zr22n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 05:45:55.652172 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.652058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45a50ce9-2dfc-46eb-aaef-ea0d195ae141" (UID: "45a50ce9-2dfc-46eb-aaef-ea0d195ae141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 05:45:55.750541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.750499    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr22n\" (UniqueName: \"kubernetes.io/projected/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-kube-api-access-zr22n\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:55.750541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.750534    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-openshift-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:55.750541 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:55.750547    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a50ce9-2dfc-46eb-aaef-ea0d195ae141-maas-api-tls\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:56.043611 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.043543    2582 generic.go:356] "Generic (PLEG): container finished" podID="45a50ce9-2dfc-46eb-aaef-ea0d195ae141" containerID="036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256" exitCode=0
Jul 17 05:45:56.043611 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.043591    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"
Jul 17 05:45:56.044028 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.043582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" event={"ID":"45a50ce9-2dfc-46eb-aaef-ea0d195ae141","Type":"ContainerDied","Data":"036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256"}
Jul 17 05:45:56.044028 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.043709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7" event={"ID":"45a50ce9-2dfc-46eb-aaef-ea0d195ae141","Type":"ContainerDied","Data":"3d26f66b7bc8304f062bf67dc3b69df37dc62587023575a555e37cde7732ad2a"}
Jul 17 05:45:56.044028 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.043734    2582 scope.go:117] "RemoveContainer" containerID="036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256"
Jul 17 05:45:56.054675 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.054652    2582 scope.go:117] "RemoveContainer" containerID="036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256"
Jul 17 05:45:56.054985 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:45:56.054967    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256\": container with ID starting with 036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256 not found: ID does not exist" containerID="036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256"
Jul 17 05:45:56.055044 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.054993    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256"} err="failed to get container status \"036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256\": rpc error: code = NotFound desc = could not find container \"036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256\": container with ID starting with 036aaa63ecc966a94f987bebb5073ed0a61389d3a9f68de5a877a7f552040256 not found: ID does not exist"
Jul 17 05:45:56.069515 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.069484    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"]
Jul 17 05:45:56.073589 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.073561    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-823cda-858f6754d5-dclb7"]
Jul 17 05:45:56.159958 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.159934    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:56.255029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.254994    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1541ebeb-10e3-4ca8-b05d-e6de59fa726e-maas-api-service-ca\") pod \"1541ebeb-10e3-4ca8-b05d-e6de59fa726e\" (UID: \"1541ebeb-10e3-4ca8-b05d-e6de59fa726e\") "
Jul 17 05:45:56.255389 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.255368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1541ebeb-10e3-4ca8-b05d-e6de59fa726e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1541ebeb-10e3-4ca8-b05d-e6de59fa726e" (UID: "1541ebeb-10e3-4ca8-b05d-e6de59fa726e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:45:56.356464 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.356416    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1541ebeb-10e3-4ca8-b05d-e6de59fa726e-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:45:56.729764 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:56.729732    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45a50ce9-2dfc-46eb-aaef-ea0d195ae141" path="/var/lib/kubelet/pods/45a50ce9-2dfc-46eb-aaef-ea0d195ae141/volumes"
Jul 17 05:45:57.050284 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:57.050182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf" event={"ID":"1541ebeb-10e3-4ca8-b05d-e6de59fa726e","Type":"ContainerDied","Data":"59360d526549edb639607d747fff1d8096731b913e528ec312d6a33341b5c84c"}
Jul 17 05:45:57.050284 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:57.050209    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"
Jul 17 05:45:57.050284 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:57.050237    2582 scope.go:117] "RemoveContainer" containerID="b00ba9932a69d06b37c62f8f73ffb6d7e4af91ca30b240d7e35fbcd4a12a304b"
Jul 17 05:45:57.067224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:57.067184    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"]
Jul 17 05:45:57.072214 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:57.072190    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf0ad9db-zqgtf"]
Jul 17 05:45:57.295996 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:57.295963    2582 scope.go:117] "RemoveContainer" containerID="6c8b270ddcc219bdc42701ee8c8742670131f24c31570c4fe76b1178ed7ec1b3"
Jul 17 05:45:58.729556 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:45:58.729523    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" path="/var/lib/kubelet/pods/1541ebeb-10e3-4ca8-b05d-e6de59fa726e/volumes"
Jul 17 05:46:11.346301 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.346248    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d874fd7c6-rcgtc_de593a03-4536-413f-b011-b930e1d849eb/authorino/0.log"
Jul 17 05:46:11.564892 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.564855    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"]
Jul 17 05:46:11.565281 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565247    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:11.565331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565284    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:11.565331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565299    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:11.565331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565307    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:11.565331 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565327    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565337    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565347    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b73d8f-f964-4b49-bb59-d5acc612e15e" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565355    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b73d8f-f964-4b49-bb59-d5acc612e15e" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565365    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565374    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565390    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a50ce9-2dfc-46eb-aaef-ea0d195ae141" containerName="maas-api"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565397    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a50ce9-2dfc-46eb-aaef-ea0d195ae141" containerName="maas-api"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565409    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565416    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:11.565476 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565477    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:11.565770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565489    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="45a50ce9-2dfc-46eb-aaef-ea0d195ae141" containerName="maas-api"
Jul 17 05:46:11.565770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565498    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:11.565770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565506    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b73d8f-f964-4b49-bb59-d5acc612e15e" containerName="revoke-keys"
Jul 17 05:46:11.565770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565517    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="502fd029-c7d6-4bd8-af1a-d52a4592e21f" containerName="revoke-keys"
Jul 17 05:46:11.565770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565526    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:11.565770 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.565534    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:11.568745 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.568728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:11.571347 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.571319    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 05:46:11.571481 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.571319    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qh8zj\""
Jul 17 05:46:11.575017 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.574993    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"]
Jul 17 05:46:11.689511 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.689417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a5a81-de67-48b8-954a-f82eb3c8d699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-t77bs\" (UID: \"610a5a81-de67-48b8-954a-f82eb3c8d699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:11.790658 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.790606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a5a81-de67-48b8-954a-f82eb3c8d699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-t77bs\" (UID: \"610a5a81-de67-48b8-954a-f82eb3c8d699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:11.791291 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.791238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a5a81-de67-48b8-954a-f82eb3c8d699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e221674-t77bs\" (UID: \"610a5a81-de67-48b8-954a-f82eb3c8d699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:11.879849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:11.879819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:12.018429 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:12.015288    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"]
Jul 17 05:46:12.021919 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:46:12.021882    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610a5a81_de67_48b8_954a_f82eb3c8d699.slice/crio-bdd59d0a055b5cce73a9e1c59ecd1c972aeeaab1522b9f1dd84bc215b70fd758 WatchSource:0}: Error finding container bdd59d0a055b5cce73a9e1c59ecd1c972aeeaab1522b9f1dd84bc215b70fd758: Status 404 returned error can't find the container with id bdd59d0a055b5cce73a9e1c59ecd1c972aeeaab1522b9f1dd84bc215b70fd758
Jul 17 05:46:12.116390 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:12.116350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" event={"ID":"610a5a81-de67-48b8-954a-f82eb3c8d699","Type":"ContainerStarted","Data":"bdd59d0a055b5cce73a9e1c59ecd1c972aeeaab1522b9f1dd84bc215b70fd758"}
Jul 17 05:46:13.122673 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:13.122636    2582 generic.go:356] "Generic (PLEG): container finished" podID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerID="84f6f18a99199b7c46088032b4695e05490bc5d4580c3cd47dd3d2c1eb9b4f2a" exitCode=6
Jul 17 05:46:13.123064 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:13.122686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" event={"ID":"610a5a81-de67-48b8-954a-f82eb3c8d699","Type":"ContainerDied","Data":"84f6f18a99199b7c46088032b4695e05490bc5d4580c3cd47dd3d2c1eb9b4f2a"}
Jul 17 05:46:13.123064 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:13.122921    2582 scope.go:117] "RemoveContainer" containerID="84f6f18a99199b7c46088032b4695e05490bc5d4580c3cd47dd3d2c1eb9b4f2a"
Jul 17 05:46:14.128453 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:14.128419    2582 generic.go:356] "Generic (PLEG): container finished" podID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerID="f99aebef745e72af92919d543aa3093af9c13540587d584d1ec385e6dbccc0b1" exitCode=6
Jul 17 05:46:14.128877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:14.128510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" event={"ID":"610a5a81-de67-48b8-954a-f82eb3c8d699","Type":"ContainerDied","Data":"f99aebef745e72af92919d543aa3093af9c13540587d584d1ec385e6dbccc0b1"}
Jul 17 05:46:14.128877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:14.128573    2582 scope.go:117] "RemoveContainer" containerID="84f6f18a99199b7c46088032b4695e05490bc5d4580c3cd47dd3d2c1eb9b4f2a"
Jul 17 05:46:14.128877 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:14.128673    2582 scope.go:117] "RemoveContainer" containerID="f99aebef745e72af92919d543aa3093af9c13540587d584d1ec385e6dbccc0b1"
Jul 17 05:46:14.129028 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:46:14.128894    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-t77bs_odh-ai-gateway-infra(610a5a81-de67-48b8-954a-f82eb3c8d699)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699"
Jul 17 05:46:15.135844 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:15.135816    2582 scope.go:117] "RemoveContainer" containerID="f99aebef745e72af92919d543aa3093af9c13540587d584d1ec385e6dbccc0b1"
Jul 17 05:46:15.136341 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:46:15.136015    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-t77bs_odh-ai-gateway-infra(610a5a81-de67-48b8-954a-f82eb3c8d699)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699"
Jul 17 05:46:17.727830 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:17.727801    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d857c78d5-qzbdb_ea8d7925-3cf2-4715-a627-62285a888f43/copy-manifests/0.log"
Jul 17 05:46:17.739649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:17.739617    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d857c78d5-qzbdb_ea8d7925-3cf2-4715-a627-62285a888f43/manager/0.log"
Jul 17 05:46:18.343719 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:18.343681    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737770-7652d_c0662ef5-0f74-4fb5-b7c4-ce20800c2031/cleanup/0.log"
Jul 17 05:46:18.574064 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:18.574040    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-4e221674-t77bs_610a5a81-de67-48b8-954a-f82eb3c8d699/revoke-keys/1.log"
Jul 17 05:46:18.792849 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:18.792821    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-938bd0-kzvbj_57d6fe0b-727d-42da-8cbb-939697dbdf82/revoke-keys/0.log"
Jul 17 05:46:18.906974 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:18.906949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-ce2966-hcmrz_9a70986f-8baa-4529-928a-c4da699b15e4/revoke-keys/0.log"
Jul 17 05:46:19.017731 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:19.017702    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-e1ab5d-4ckns_85d3aa4a-f08a-4822-89be-dad8af281b01/revoke-keys/0.log"
Jul 17 05:46:19.126111 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:19.126039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-63fb52-pclx5_cf4fd38e-9dbd-44d8-b73e-87505e089eab/revoke-keys/0.log"
Jul 17 05:46:19.238213 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:19.238186    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-857d512b-mvdd5_1eed1d48-d972-4463-b08b-9db904e1c353/revoke-keys/0.log"
Jul 17 05:46:19.570729 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:19.570702    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-823cda-g8txn_71b73d8f-f964-4b49-bb59-d5acc612e15e/revoke-keys/0.log"
Jul 17 05:46:20.976599 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:20.976569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d874fd7c6-rcgtc_de593a03-4536-413f-b011-b930e1d849eb/authorino/0.log"
Jul 17 05:46:22.530714 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:22.530690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d686d4c64-tcwx9_e1d59511-6785-415f-b095-6dc3dd7031d5/kube-auth-proxy/0.log"
Jul 17 05:46:22.901002 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:22.900923    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fnxlv_0b645dd0-44aa-4aaa-94ff-ac84a6a361d0/payload-processing/0.log"
Jul 17 05:46:23.692394 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:23.692362    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db9llfs_f4881ea1-ca67-462c-b5d7-23807a56d489/main/0.log"
Jul 17 05:46:23.798630 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:23.798604    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-td42d_6faecf39-ede8-428b-b1d3-a4ebffc5ecfc/main/0.log"
Jul 17 05:46:27.725201 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:27.725166    2582 scope.go:117] "RemoveContainer" containerID="f99aebef745e72af92919d543aa3093af9c13540587d584d1ec385e6dbccc0b1"
Jul 17 05:46:28.193247 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:28.193211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" event={"ID":"610a5a81-de67-48b8-954a-f82eb3c8d699","Type":"ContainerStarted","Data":"13657c349067bf5c13d2a47fcdcd81c51136d03272c80f0bdc837395906f08d1"}
Jul 17 05:46:28.209110 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:28.209057    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" podStartSLOduration=17.209039514 podStartE2EDuration="17.209039514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:28.207381563 +0000 UTC m=+3391.944443898" watchObservedRunningTime="2026-07-17 05:46:28.209039514 +0000 UTC m=+3391.946101850"
Jul 17 05:46:29.199303 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.199249    2582 generic.go:356] "Generic (PLEG): container finished" podID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerID="13657c349067bf5c13d2a47fcdcd81c51136d03272c80f0bdc837395906f08d1" exitCode=6
Jul 17 05:46:29.199732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.199328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" event={"ID":"610a5a81-de67-48b8-954a-f82eb3c8d699","Type":"ContainerDied","Data":"13657c349067bf5c13d2a47fcdcd81c51136d03272c80f0bdc837395906f08d1"}
Jul 17 05:46:29.199732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.199392    2582 scope.go:117] "RemoveContainer" containerID="f99aebef745e72af92919d543aa3093af9c13540587d584d1ec385e6dbccc0b1"
Jul 17 05:46:29.199732 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.199688    2582 scope.go:117] "RemoveContainer" containerID="13657c349067bf5c13d2a47fcdcd81c51136d03272c80f0bdc837395906f08d1"
Jul 17 05:46:29.199905 ip-10-0-140-139 kubenswrapper[2582]: E0717 05:46:29.199885    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4e221674-t77bs_odh-ai-gateway-infra(610a5a81-de67-48b8-954a-f82eb3c8d699)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699"
Jul 17 05:46:29.218879 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.218852    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"]
Jul 17 05:46:29.447934 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.447898    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvgtg/must-gather-db52q"]
Jul 17 05:46:29.448202 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.448189    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:29.448246 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.448203    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:29.448246 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.448219    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:29.448246 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.448225    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:29.448388 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.448290    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1541ebeb-10e3-4ca8-b05d-e6de59fa726e" containerName="revoke-keys"
Jul 17 05:46:29.448388 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.448298    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22177f62-ffc1-40cc-8309-0d700c1cc9fa" containerName="revoke-keys"
Jul 17 05:46:29.452523 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.452473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.455272 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.455233    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvgtg\"/\"openshift-service-ca.crt\""
Jul 17 05:46:29.455429 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.455368    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fvgtg\"/\"default-dockercfg-78785\""
Jul 17 05:46:29.455535 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.455480    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvgtg\"/\"kube-root-ca.crt\""
Jul 17 05:46:29.464041 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.464019    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvgtg/must-gather-db52q"]
Jul 17 05:46:29.539484 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.539439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jlx\" (UniqueName: \"kubernetes.io/projected/9c5468a4-4b31-44fe-8f82-24727e95bb4f-kube-api-access-p9jlx\") pod \"must-gather-db52q\" (UID: \"9c5468a4-4b31-44fe-8f82-24727e95bb4f\") " pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.539662 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.539492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c5468a4-4b31-44fe-8f82-24727e95bb4f-must-gather-output\") pod \"must-gather-db52q\" (UID: \"9c5468a4-4b31-44fe-8f82-24727e95bb4f\") " pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.640152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.640106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jlx\" (UniqueName: \"kubernetes.io/projected/9c5468a4-4b31-44fe-8f82-24727e95bb4f-kube-api-access-p9jlx\") pod \"must-gather-db52q\" (UID: \"9c5468a4-4b31-44fe-8f82-24727e95bb4f\") " pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.640152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.640162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c5468a4-4b31-44fe-8f82-24727e95bb4f-must-gather-output\") pod \"must-gather-db52q\" (UID: \"9c5468a4-4b31-44fe-8f82-24727e95bb4f\") " pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.640504 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.640488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9c5468a4-4b31-44fe-8f82-24727e95bb4f-must-gather-output\") pod \"must-gather-db52q\" (UID: \"9c5468a4-4b31-44fe-8f82-24727e95bb4f\") " pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.654750 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.654729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jlx\" (UniqueName: \"kubernetes.io/projected/9c5468a4-4b31-44fe-8f82-24727e95bb4f-kube-api-access-p9jlx\") pod \"must-gather-db52q\" (UID: \"9c5468a4-4b31-44fe-8f82-24727e95bb4f\") " pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.761408 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.761329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvgtg/must-gather-db52q"
Jul 17 05:46:29.892649 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.892556    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvgtg/must-gather-db52q"]
Jul 17 05:46:29.894760 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:46:29.894731    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c5468a4_4b31_44fe_8f82_24727e95bb4f.slice/crio-f24b967db20b4bd01a457d4c0461412dc213479d9d746c9f8b66caaf2e285b7b WatchSource:0}: Error finding container f24b967db20b4bd01a457d4c0461412dc213479d9d746c9f8b66caaf2e285b7b: Status 404 returned error can't find the container with id f24b967db20b4bd01a457d4c0461412dc213479d9d746c9f8b66caaf2e285b7b
Jul 17 05:46:29.896500 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:29.896483    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 05:46:30.206842 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:30.206815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvgtg/must-gather-db52q" event={"ID":"9c5468a4-4b31-44fe-8f82-24727e95bb4f","Type":"ContainerStarted","Data":"f24b967db20b4bd01a457d4c0461412dc213479d9d746c9f8b66caaf2e285b7b"}
Jul 17 05:46:30.330692 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:30.330664    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:30.448022 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:30.447990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a5a81-de67-48b8-954a-f82eb3c8d699-maas-api-service-ca\") pod \"610a5a81-de67-48b8-954a-f82eb3c8d699\" (UID: \"610a5a81-de67-48b8-954a-f82eb3c8d699\") "
Jul 17 05:46:30.448416 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:30.448395    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610a5a81-de67-48b8-954a-f82eb3c8d699-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "610a5a81-de67-48b8-954a-f82eb3c8d699" (UID: "610a5a81-de67-48b8-954a-f82eb3c8d699"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 05:46:30.549440 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:30.549349    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610a5a81-de67-48b8-954a-f82eb3c8d699-maas-api-service-ca\") on node \"ip-10-0-140-139.ec2.internal\" DevicePath \"\""
Jul 17 05:46:31.214755 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.214707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs" event={"ID":"610a5a81-de67-48b8-954a-f82eb3c8d699","Type":"ContainerDied","Data":"bdd59d0a055b5cce73a9e1c59ecd1c972aeeaab1522b9f1dd84bc215b70fd758"}
Jul 17 05:46:31.215239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.214792    2582 scope.go:117] "RemoveContainer" containerID="13657c349067bf5c13d2a47fcdcd81c51136d03272c80f0bdc837395906f08d1"
Jul 17 05:46:31.215239 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.214923    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"
Jul 17 05:46:31.217305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.217199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvgtg/must-gather-db52q" event={"ID":"9c5468a4-4b31-44fe-8f82-24727e95bb4f","Type":"ContainerStarted","Data":"7f29729785dfccf3929a555e09d9682764790085d0186aaecdbf9191d4dccee2"}
Jul 17 05:46:31.217305 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.217247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvgtg/must-gather-db52q" event={"ID":"9c5468a4-4b31-44fe-8f82-24727e95bb4f","Type":"ContainerStarted","Data":"b9e0dfd1288b5243cb9378d6e3ab5e54b09c205d51a18cb345398509cd58bd1f"}
Jul 17 05:46:31.234370 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.234297    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvgtg/must-gather-db52q" podStartSLOduration=1.38784752 podStartE2EDuration="2.234250277s" totalImagesPullingTime="846.402757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:31.232923296 +0000 UTC m=+3394.969985632" watchObservedRunningTime="2026-07-17 05:46:31.234250277 +0000 UTC m=+3394.971312613"
Jul 17 05:46:31.246506 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.246467    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"]
Jul 17 05:46:31.252029 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:31.251997    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e221674-t77bs"]
Jul 17 05:46:32.305401 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:32.305333    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-drpqq_b5be18d5-6b55-42fa-8669-03ac30ab7f33/global-pull-secret-syncer/0.log"
Jul 17 05:46:32.431565 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:32.431521    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d9x2p_94cabee3-1dc5-4b2f-97be-f5d095b26776/konnectivity-agent/0.log"
Jul 17 05:46:32.545532 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:32.545500    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-139.ec2.internal_71c35c8b98f8ea365483d4a890e41d1a/haproxy/0.log"
Jul 17 05:46:32.730061 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:32.730019    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" path="/var/lib/kubelet/pods/610a5a81-de67-48b8-954a-f82eb3c8d699/volumes"
Jul 17 05:46:37.403478 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:37.403441    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d874fd7c6-rcgtc_de593a03-4536-413f-b011-b930e1d849eb/authorino/0.log"
Jul 17 05:46:39.710645 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:39.710609    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffnhs_61b8012a-a333-4b4e-b963-859b3513e8f2/node-exporter/0.log"
Jul 17 05:46:39.732477 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:39.732425    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffnhs_61b8012a-a333-4b4e-b963-859b3513e8f2/kube-rbac-proxy/0.log"
Jul 17 05:46:39.755862 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:39.755792    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffnhs_61b8012a-a333-4b4e-b963-859b3513e8f2/init-textfile/0.log"
Jul 17 05:46:42.526075 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526033    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"]
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526494    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526514    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526534    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526542    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526552    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526561    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526647    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.526697 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.526660    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="610a5a81-de67-48b8-954a-f82eb3c8d699" containerName="revoke-keys"
Jul 17 05:46:42.532282 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.532237    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.537189 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.537152    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"]
Jul 17 05:46:42.679891 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.679847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-lib-modules\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.680216 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.680197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-podres\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.680411 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.680392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-proc\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.680593 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.680562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-sys\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.680746 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.680730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blch\" (UniqueName: \"kubernetes.io/projected/0764442b-af82-4833-9ece-ed7733380f57-kube-api-access-6blch\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.781824 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.781732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-sys\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782013 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.781835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blch\" (UniqueName: \"kubernetes.io/projected/0764442b-af82-4833-9ece-ed7733380f57-kube-api-access-6blch\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782013 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.781867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-sys\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782228 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.782204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-lib-modules\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782391 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.782371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-lib-modules\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782498 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.782478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-podres\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782571 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.782399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-podres\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782571 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.782539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-proc\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.782680 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.782631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0764442b-af82-4833-9ece-ed7733380f57-proc\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.789671 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.789632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blch\" (UniqueName: \"kubernetes.io/projected/0764442b-af82-4833-9ece-ed7733380f57-kube-api-access-6blch\") pod \"perf-node-gather-daemonset-swv54\" (UID: \"0764442b-af82-4833-9ece-ed7733380f57\") " pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:42.847667 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:42.847627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:43.034832 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:43.034695    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"]
Jul 17 05:46:43.043207 ip-10-0-140-139 kubenswrapper[2582]: W0717 05:46:43.043174    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0764442b_af82_4833_9ece_ed7733380f57.slice/crio-57c6ef1ea88b43d49e740574ade2a01cf500798dade30d042c699d34629e8e0a WatchSource:0}: Error finding container 57c6ef1ea88b43d49e740574ade2a01cf500798dade30d042c699d34629e8e0a: Status 404 returned error can't find the container with id 57c6ef1ea88b43d49e740574ade2a01cf500798dade30d042c699d34629e8e0a
Jul 17 05:46:43.302647 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:43.302565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54" event={"ID":"0764442b-af82-4833-9ece-ed7733380f57","Type":"ContainerStarted","Data":"6c47d41cb44776ad4422fb8b7bd6253c34d9595b061628679a3b635eac4c11db"}
Jul 17 05:46:43.302647 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:43.302625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54" event={"ID":"0764442b-af82-4833-9ece-ed7733380f57","Type":"ContainerStarted","Data":"57c6ef1ea88b43d49e740574ade2a01cf500798dade30d042c699d34629e8e0a"}
Jul 17 05:46:43.302856 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:43.302710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:43.319668 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:43.319619    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54" podStartSLOduration=1.319602009 podStartE2EDuration="1.319602009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:43.318599274 +0000 UTC m=+3407.055661633" watchObservedRunningTime="2026-07-17 05:46:43.319602009 +0000 UTC m=+3407.056664343"
Jul 17 05:46:44.548478 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:44.548452    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4pnz8_642d9a54-09c6-4827-9500-18ae6b91a4e5/dns/0.log"
Jul 17 05:46:44.570790 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:44.570769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4pnz8_642d9a54-09c6-4827-9500-18ae6b91a4e5/kube-rbac-proxy/0.log"
Jul 17 05:46:44.731167 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:44.731066    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6s8kt_90ebd0bd-0591-47a7-b42f-79c949d53fe7/dns-node-resolver/0.log"
Jul 17 05:46:45.278169 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:45.278137    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8677c6c9b4-wrjqg_ec735f34-6ae1-41af-ba35-692446d958f2/registry/0.log"
Jul 17 05:46:45.345323 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:45.345229    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmd9s_3efb5631-c960-43ec-86b8-3038e6486013/node-ca/0.log"
Jul 17 05:46:46.515669 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:46.515631    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d686d4c64-tcwx9_e1d59511-6785-415f-b095-6dc3dd7031d5/kube-auth-proxy/0.log"
Jul 17 05:46:46.672932 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:46.672896    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fnxlv_0b645dd0-44aa-4aaa-94ff-ac84a6a361d0/payload-processing/0.log"
Jul 17 05:46:47.414197 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:47.414169    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmwff_db937e1e-b076-4752-8f84-4e7704ace47f/serve-healthcheck-canary/0.log"
Jul 17 05:46:48.014289 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:48.014246    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbzk2_40def634-4715-4391-ad45-4c6acb03ce21/kube-rbac-proxy/0.log"
Jul 17 05:46:48.033699 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:48.033668    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbzk2_40def634-4715-4391-ad45-4c6acb03ce21/exporter/0.log"
Jul 17 05:46:48.054443 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:48.054407    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbzk2_40def634-4715-4391-ad45-4c6acb03ce21/extractor/0.log"
Jul 17 05:46:49.318098 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:49.318067    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fvgtg/perf-node-gather-daemonset-swv54"
Jul 17 05:46:50.714704 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:50.714673    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d857c78d5-qzbdb_ea8d7925-3cf2-4715-a627-62285a888f43/manager/0.log"
Jul 17 05:46:50.731236 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:50.731204    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d857c78d5-qzbdb_ea8d7925-3cf2-4715-a627-62285a888f43/copy-manifests/0.log"
Jul 17 05:46:52.090087 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:52.090057    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-559bdd8b4c-6crsw_b6025461-9594-48ae-a90d-b7a2779befe8/manager/0.log"
Jul 17 05:46:52.143249 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:52.143220    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gpmnf_e3c3e2c4-757a-4710-a4ad-9f973a1225f4/openshift-lws-operator/0.log"
Jul 17 05:46:53.295356 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:53.295327    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d686d4c64-tcwx9_e1d59511-6785-415f-b095-6dc3dd7031d5/kube-auth-proxy/0.log"
Jul 17 05:46:53.449423 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:53.449395    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fnxlv_0b645dd0-44aa-4aaa-94ff-ac84a6a361d0/payload-processing/0.log"
Jul 17 05:46:54.908966 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:54.908911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d686d4c64-tcwx9_e1d59511-6785-415f-b095-6dc3dd7031d5/kube-auth-proxy/0.log"
Jul 17 05:46:55.042210 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:55.042177    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fnxlv_0b645dd0-44aa-4aaa-94ff-ac84a6a361d0/payload-processing/0.log"
Jul 17 05:46:57.218863 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:57.218833    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-vrhtt_8ee5eef8-1714-4f08-8280-eaa0f6839005/migrator/0.log"
Jul 17 05:46:57.239152 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:57.239118    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-vrhtt_8ee5eef8-1714-4f08-8280-eaa0f6839005/graceful-termination/0.log"
Jul 17 05:46:58.812909 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.812881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/kube-multus-additional-cni-plugins/0.log"
Jul 17 05:46:58.834224 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.834197    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/egress-router-binary-copy/0.log"
Jul 17 05:46:58.855621 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.855562    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/cni-plugins/0.log"
Jul 17 05:46:58.876744 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.876688    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/bond-cni-plugin/0.log"
Jul 17 05:46:58.896751 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.896730    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/routeoverride-cni/0.log"
Jul 17 05:46:58.920021 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.919996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/whereabouts-cni-bincopy/0.log"
Jul 17 05:46:58.941353 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:58.941331    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d282m_2c6552ab-867d-4bc6-8958-d12371149d7d/whereabouts-cni/0.log"
Jul 17 05:46:59.074140 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:59.074058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qf4nq_ce414902-197a-417e-8605-a88717e915ec/kube-multus/0.log"
Jul 17 05:46:59.099378 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:59.099349    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9nwkj_cc964d46-6df9-4848-8b80-ce5423eab530/network-metrics-daemon/0.log"
Jul 17 05:46:59.116377 ip-10-0-140-139 kubenswrapper[2582]: I0717 05:46:59.116351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9nwkj_cc964d46-6df9-4848-8b80-ce5423eab530/kube-rbac-proxy/0.log"